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Why  GAO  Did  This  Study 

DOD  relies  on  innovative  technologies 
to  ensure  the  superiority  of  its  weapon 
systems  and  planned  to  invest  about 
$12.5  billion  in  fiscal  year  2017  to 
achieve  this  aim.  Recently,  DOD’s 
leadership  role  in  fostering  innovation 
has  been  supplanted  by  the 
commercial  sector.  This  has  changed 
DOD’s  approach  to  technology 
development  by  relying  more  on 
commercial  innovation. 

Conference  Report  112-329  included  a 
provision  for  GAO  to  review  DOD’s 
S&T  enterprise.  This  report  assesses 
(1)  the  practices  leading  companies 
employ  to  manage  technology 
development  and  (2)  the  extent  to 
which  DOD  can  incorporate  these 
practices  into  its  own.  GAO  interviewed 
eight  large,  profitable,  leading 
technology  companies  (Amazon,  Dow 
Chemical,  Honeywell,  General  Motors, 
IBM,  Qualcomm,  Siemens  AG,  and 
Valvoline)  to  identify  practices  they 
used  to  manage,  prioritize,  and  assess 
their  technology  portfolios.  GAO  also 
met  with  DOD  organizations  that 
manage  and  execute  S&T  funds  to 
identify  their  practices. 

What  GAO  Recommends 

GAO  recommends  that  DOD 
annually  define  and  assess  the  mix 
of  innovation  investments  and  define, 
in  policy  or  guidance,  an  S&T 
management  framework  that 
comprehensively  employs  leading 
commercial  practices.  DOD  did  not 
agree  with  the  recommendations, 
citing  its  ongoing  deliberations  on  the 
new  USD  R&E’s  role,  but  did  identify 
some  planned  actions.  GAO  believes 
its  recommendations  are  valid  as 
discussed  in  the  report. 

View  GAO-17-499.  For  more  information, 
contact  Mike  Sullivan  at  (202)  512-4841  or 
sullivanm@gao.gov. 


What  GAO  Found 

The  eight  leading  companies  whose  practices  GAO  assessed  take  a  disciplined 
approach  to  organizing  and  executing  their  technology  development  activities  by 
grouping  them  into  two  portfolios:  incremental  and  disruptive,  as  shown  in  the 
figure.  Incremental  development  improves  product  lines  whereas  disruptive 
development  is  for  riskier  innovative  and  potentially  market-shifting  technologies. 

Commercial  Model  Ensures  Investments  in  Incremental  and  Disruptive  Innovation 

Strategic  direction  for  technology  development 

Leadership  defines  the  vision  for  the  company,  sets  technology 
development  budget,  and  provides  direction  on  major  projects 


Disruptive  technology 
development  portfolio 

Discretionary  funding  is  set  aside 
for  developing  futuristic,  but 
potentially  risky,  technologies  that 
might  be  integrated  into  new  or 
existing  products 

Source:  GAO  analysis  of  leading  company  R&D  practices,  )  GAO-17-499 _ 

By  separating  these  two  portfolios,  companies  reported  that  they  could  promote 
existing  product  lines  in  the  short  term  while  exploring  opportunities  to  remain 
competitive  in  the  long  term,  and  mitigate  the  financial  risk  associated  with 
disruptive  technology  development.  Moreover,  GAO  found  that  leading 
companies  also  ensure  technologies  will  be  relevant  in  the  marketplace  by 
engaging  a  wide  range  of  internal  stakeholders.  These  companies  also  reported 
that  they  gain  leadership  buy-in  by  prototyping  technologies  before  committing  to 
further  development  and  product  integration. 

While  some  Department  of  Defense  (DOD)  practices  closely  mirror  those  of  the 
companies  GAO  reviewed,  DOD's  ability  to  adopt  leading  commercial  practices 
in  its  approach  to  managing  science  and  technology  (S&T)  investments  is  limited 
by  its  funding  policies  and  culture.  Unlike  the  companies  GAO  reviewed,  DOD 
leadership  does  not  provide  guidance  on  or  assess  the  mix  of  incremental  and 
disruptive  innovation.  As  a  result,  officials  reported  that  DOD  labs  struggle  to  find 
the  right  balance  between  these  investment  areas.  Under  DOD’s  budget  policy, 
projects  are  planned  up  to  2  years  in  advance,  which  can  slow  innovation  and 
limit  lab  directors’  autonomy  as  compared  to  companies.  Congress  has  provided 
a  means  for  lab  directors  to  initiate  work  outside  of  this  lengthy  process,  but  it 
has  not  been  fully  utilized.  Additionally,  responsibilities  for  technology  versus 
product  development  also  contribute  to  a  culture  that  discourages  collaboration 
and  limits  labs’  ability  to  prototype.  Yet  these  issues  are  not  insurmountable,  as 
pockets  of  each  military  department  have  demonstrated,  such  as  through  recent 
efforts  to  expand  advanced  prototyping  in  the  labs.  Further,  Congress  has 
required  that  by  February  2018  DOD  create  a  new  Under  Secretary  of  Defense 
for  Research  and  Engineering  (USD(R&E)),  which  will  be  charged  with 
developing  policies  to  improve  innovation.  This  position  creates  an  opportunity  to 
develop  policies  that  further  promote  adoption  of  leading  commercial  practices. 
- United  States  Government  Accountability  Office 


Incremental  technology 
development  portfolio 

Efforts  directed  toward  upgrading 
existing  products  and  technologies, 
or  for  creating  new  products  that  fit 
within  the  company’s  strategic  plan 
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Abbreviations 


ASD(R&E) 

Assistant  Secretary  of  Defense  for  Research  and 
Engineering 

BA 

budget  activity 

BES 

budget  estimate  submission 

CEO 

Chief  Executive  Officer 

COI 

Community  of  Interest 

CTO 

Chief  Technology  Officer 

DARPA 

Defense  Advanced  Research  Projects  Agency 

DOD 

Department  of  Defense 

IBM 

International  Business  Machines 

NRL 

Naval  Research  Laboratory 

NDAA 

National  Defense  Authorization  Act 

OSD 

Office  of  the  Secretary  of  Defense 

POM 

program  objective  memorandum 

PPBE 

planning,  programming,  budgeting,  and  execution 

R&D 

research  and  development 

RDT&E 

research,  development,  test,  and  evaluation 

RPED 

Rapid  Prototyping,  Experimentation,  and  Demonstration 

S&T 

science  and  technology 

USD(R&E) 

Under  Secretary  of  Defense  for  Research  and  Engineering 
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GAO 

441  G  St.  N.W. 
Washington,  DC  20548 


U.S.  GOVERNMENT  ACCOUNTABILITY  OFFICE 


June  29,  2017 
Congressional  Committees 

The  Department  of  Defense  (DOD)  relies  on  the  technological  superiority 
of  its  weapon  systems  and  armed  forces  to  protect  U.S.  interests  at  home 
and  abroad.  This  technological  superiority  is  being  challenged  by 
potential  adversaries  in  ways  not  seen  since  the  Cold  War.  To  counter 
these  challenges,  DOD’s  science  and  technology  (S&T)  community  is 
charged  with  identifying  and  maturing  new  technologies  and  making  them 
available  to  DOD’s  acquisition  community  for  integration  into  weapon 
systems.  In  the  President’s  fiscal  year  2017  budget  submission,  DOD 
requested  approximately  $12.5  billion  for  S&T  activities  aimed  at 
developing  technologies  that  meet  both  the  short-term  and  long-term 
needs  of  current  and  future  warfighters.  In  recent  years,  DOD  has 
reported  that  its  leadership  role  in  fostering  innovation  has  been 
supplanted  by  the  commercial  sector,  which  has  changed  the  paradigm 
for  DOD  S&T.  DOD  now  relies  increasingly  on  the  innovations 
commercial  industry  produces  to  guide  its  own  S&T  investments. 

The  conference  report  112-329  accompanying  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2012  included  a  provision  for  us  to 
review  DOD’s  S&T  enterprise,  including  its  interactions  with  industry, 
investment  strategies,  technology  development  methods,  and  transition 
activities — the  process  of  migrating  new  technologies  from  the  research 
environment  to  military  users.1  We  previously  issued  a  report  in  March 
2013  to  address  the  technology  transition  component  of  this  provision.2  In 
this  report,  we  analyze  DOD’s  approach  to  managing  technology 
development  as  compared  to  best  practices  that  leading  companies  use. 
Specifically,  we  identified  (1)  practices  that  selected  leading  companies 
use  to  manage  their  S&T  portfolios  and  programs,  and  (2)  the  extent  to 
which  DOD  can  employ  these  practices  within  its  S&T  enterprise, 
including  any  barriers  to  adopting  such  practices. 


^.R.  Rep.  No.  112-329,  at  719  (2011). 

2GAO,  Defense  Technology  Development:  Technology  Transition  Programs  Support 
Military  Users ,  but  Opportunities  Exist  to  Improve  Measurement  of  Outcomes , 
GAO-13-286  (Washington,  D.C.:  Mar.  7,  2013). 


Page  1 


GAO-17-499  Innovation  Best  Practices 


To  identify  the  practices  that  leading  companies  use  to  manage  their  S&T 
portfolios  and  programs,  we  selected  eight  companies  to  include  as  case 
studies  in  our  review.  These  companies  were 

•  Amazon.com, 

•  Dow  Chemical  Company, 

•  Honeywell  International,  Inc., 

•  General  Motors  Co., 

•  International  Business  Machines  (IBM)  Corporation, 

•  Qualcomm, 

•  Siemens  AG,  and 

•  Valvoline. 

We  selected  these  companies  primarily  on  the  basis  of  their  having 
received  awards  or  other  recognition  for  technology  innovation  since 
2014,  as  well  as  factors  such  as  profitability  and  industry  type.  All  of  the 
companies  we  selected  were  profitable,  large  Fortune  500-listed 
companies,  or  were  owned  by  those  companies.  For  each  of  the 
companies,  we  interviewed  senior  management  officials  and  other 
representatives  knowledgeable  about  research  and  development  (R&D), 
which  are  technology  development  activities  equivalent  to  S&T 
development  activities  at  DOD.  We  collected  documentation,  when 
available,  and  conducted  semi-structured  interviews  to  gather  consistent 
information  about  processes  and  practices  these  selected  companies 
used  to  manage  technology  development.  In  particular,  we  obtained 
information  on  their  (1)  organizational  structures  and  management 
cultures,  (2)  R&D  portfolio  management  and  investment  strategies,  (3) 
R&D  project  management  practices,  and  (4)  their  technology  transition 
processes,  including  when  transitions  occur,  what  organizations  are 
involved,  and  how  technologies  are  further  funded.  We  synthesized  each 
company’s  processes  and  created  summary  documents,  which  the 
companies  then  reviewed  for  accuracy  and  completeness  to  validate  our 
assessment  of  their  specific  practices.  Using  this  validated  information, 
we  identified  the  practices  that  were  consistent  among  the  selected 
companies  and  which  company  representatives  considered  key  to 
promoting  innovation.  We  also  presented  our  analysis  of  these  leading 
practices  to  senior  DOD  S&T  executives  within  the  Office  of  the  Secretary 
of  Defense,  the  military  services,  and  other  defense  research 
organizations  to  obtain  their  views  on  the  practices. 
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To  identify  the  extent  to  which  DOD  can  employ  practices  that  leading 
companies  use  to  manage  S&T  investments,  we  met  with  officials  from 
across  the  DOD  S&T  enterprise,  including  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Research  and  Engineering  (ASD(R&E)),  the 
three  military  departments’  laboratories,  and  other  defense  research 
organizations  that  execute  S&T  funds.  We  conducted  semi-structured 
interviews  to  gather  consistent  information  about  processes  and  practices 
these  organization  used  to  manage  S&T  development.  In  particular,  we 
discussed  their  (1)  organizational  structure  and  management  culture,  (2) 
S&T  portfolio  management  and  investment  strategy,  (3)  S&T  project 
management  practices,  and  (4)  their  technology  transition  process.  From 
these,  we  identified  the  existing  practices  in  S&T  management  being 
used  throughout  DOD.  We  compared  and  contrasted  those  practices  with 
those  identified  through  our  meetings  with  selected  leading  companies  to 
determine  the  extent  to  which  DOD  is  employing  commercial  practices. 
We  reviewed  relevant  policies  and  regulations  used  by  DOD  S&T 
organizations  to  manage  the  S&T  enterprise.  To  further  our 
understanding  of  the  S&T  management  practices  being  used  at  the 
military  department  labs,  we  reviewed  at  least  two  recent  S&T  projects  by 
each  of  these  labs.  These  projects  were  identified  by  lab  officials  and 
included  projects  deemed  successful  as  well  as  ones  identified  as 
unsuccessful.  Finally,  we  hosted  a  forum  in  December  2016  that  included 
12  DOD  S&T  executives,  including  those  associated  with  the 
organizations  we  met  with  during  our  review,  to  identify  what  opportunities 
exist  within  DOD  to  adopt  selected  leading  companies’  practices,  as  well 
as  any  obstacles  that  may  preclude  DOD  from  fully  adopting  these 
practices.  Appendix  I  provides  additional  detail  on  our  objectives,  scope, 
and  methodology. 

We  conducted  this  performance  audit  from  February  2016  to  June  2017 
in  accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


Background 


DOD’s  S&T  community — including  research  laboratories,  test  facilities, 
industry,  and  academia — conducts  initial  research,  development,  and 
testing  of  new  technologies  to  improve  military  operations  and  ensure 
technological  superiority  over  potential  adversaries.  Key  expectations 
DOD  places  on  its  S&T  community  include  the  following: 
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•  expand  scientific  knowledge  and  investigate  technologies  that  may 
provide  new  warfighting  capabilities, 

•  anticipate  technological  needs  for  an  uncertain  future,  and 

•  produce  relevant  and  feasible  technologies  that  can  transition  into 
weapon  system  programs  or  go  directly  to  the  warfighter  in  the  field. 

As  a  result,  some  investments  focus  on  conducting  research  to  generate 
scientific  knowledge,  exploring  new  technologies,  demonstrating  the 
feasibility  of  a  technology  concept,  and  pursuing  other  science  and 
technology  endeavors.  We  have  previously  reported  that  the  challenge  is 
finding  the  right  balance  between  developing  breakthrough  or  “disruptive” 
technologies — those  considered  to  be  innovative — and  investing  in 
moderate,  “incremental”  technology  enhancements.3 

Figure  1  below  provides  a  notional  picture  of  how  DOD’s  S&T  community 
manages  technology  investment,  development,  and  transition  to  a  user. 


Figure  1 :  Department  of  Defense  Technology  Management  Process 


Technology  Investment 


Technology  Development 


Product  Development 


Technology  shelved 
but  available  for  use 


Technology 
integrated  into 
acquisition 
program  or  fielded 


Source:  GAO  analysis  of  Department  of  Defense  science  &  technology  management  process.  |  GAO-17-4S9 


Following  technology  development,  DOD’s  acquisition  community 
manages  the  next  phase,  product  development,  in  which  technologies  are 
further  advanced  and  system  development  begins.  DOD  has  long 
reported  the  existence  of  a  chasm  between  its  S&T  community  and  the 
acquisition  community,  which  often  precludes  effective  transitioning  of 


3GAO-1 3-286. 
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technologies  out  of  the  S&T  environment  into  weapon  systems.  In  a 
series  of  reports,  we  found  that  technologies  may  not  leave  the  lab 
because  their  potential  has  not  been  adequately  demonstrated  or 
recognized,  acquisition  programs  may  be  unwilling  to  fund  final  stages  of 
development,  or  private  industry  chooses  to  develop  the  technologies 
itself.4  Further,  we  found  that  the  acquisition  community  frequently 
integrates  technologies  too  early  and  takes  on  the  task  of  maturing 
technologies — an  activity  that  is  the  primary  responsibility  of  the  S&T 
community — at  the  start  of  an  acquisition  program.  These  challenges,  in 
part,  contribute  to  cost  growth,  schedule  delays,  and  performance 
shortfalls  that  we  have  frequently  found  and  reported  on  in  DOD  weapon 
programs.5 

DOD  funds  technology  and  product  development  activities  under  its 
research,  development,  test,  and  evaluation  (RDT&E)  budget,  which  DOD 
groups  into  seven  budget  activity  categories  for  its  annual  budget 
estimates.  The  categories  follow  a  mostly  sequential  path  for  developing 
technologies  from  basic  research  to  operational  system  development,  as 
is  shown  in  figure  2.  The  first  three  budget  activity  categories  generally 
represent  activities  undertaken  by  DOD’s  S&T  enterprise  to  advance 
research  and  develop  technology,  while  the  remaining  budget  activity 
categories  are  typically  associated  with  product  development  for 
acquisition  programs.  See  Appendix  II  for  a  description  of  each  budget 
activity. 


4See,  e.g.,  GAO-13-286,  GAO,  Best  Practices:  Stronger  Practices  Needed  to  Improve 
DOD  Technology  Transition  Processes,  GAO-06-883  (Washington,  D.C.:  Sept.  14,  2006); 
Small  Business  Innovation  Research:  DOD's  Program  Supports  Weapon  Systems,  but 
Lacks  Comprehensive  Data  on  Technology  Transition  Outcomes,  GAO-14-96 
(Washington,  D.C.:  Dec.  20,  2013);  and  Defense  Advanced  Research  Projects  Agency: 
Key  Factors  Drive  Transition  of  Technologies,  but  Better  Training  and  Data  Dissemination 
Can  Increase  Success,  GAO-16-5  (Washington,  D.C.:  Nov.  18,  2015). 

5See,  e.g.,  GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapons  Programs, 
GAO-16-329SP  (Washington,  D.C.:  Mar.  31, 2016);  and  Defense  Acquisitions: 
Assessments  of  Selected  Weapons  Programs,  GAO-1 7-333SP  (Washington,  D.C.:  Mar. 
30,2017). 
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Figure  2:  Department  of  Defense  (DOD)  Research,  Development,  Test,  and  Evaluation  Budget  Activities 


Source:  GAO  analysis  of  DOD  documents  |  GAO-1 7-499 
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Leading  Companies 
Foster  Innovation  by 
Investing  in  Both 
Near-  and  Long-Term 
R&D  and  Taking 
Steps  to  Prove  New 
Technologies  Work 


Selected  leading  companies  that  we  reviewed  follow  six  key  practices  that 
together  reflect  a  disciplined  approach  to  managing  their  R&D  activities — 
those  akin  to  DOD’s  S&T  activities.  First,  they  define  their  corporate 
strategy  by  identifying  desired  markets.  Next,  they  invest  in  technology 
programs  to  penetrate  those  desired  markets.  Effective  management  of 
these  portfolios  requires  balancing  investments  between  two  types  of 
R&D  efforts:  incremental  R&D,  which  is  tied  to  near-term  products;  and 
disruptive  R&D,  which  is  intended  to  deliver  innovative  technologies  that 
can  provide  longer-term  growth.  According  to  company  representatives, 
this  balance  is  driven  by  the  business  imperative  of  sustaining  current 
markets  while  also  developing  future  ones.  Leading  companies  align  their 
goals  for  incremental  technology  development  with  product  development, 
while  also  providing  independent  paths  for  developing  disruptive 
technologies  not  tied  to  product  development.  In  addition,  these 
companies  identify  stakeholders  outside  the  scientific  realm  and 
collaborate  extensively  with  them  to  ensure  that  technologies  are  relevant 
and  can  be  efficiently  integrated  into  marketable  products.  Among  the  key 
R&D  stakeholders  are  representatives  from  the  business  units  who  are 
responsible  for  identifying  customer  needs  and  getting  products  to 
market.  They  also  scale  the  rigor  in  project  oversight  based  on  the 
amounts  of  time  and  money  invested.  Nonetheless,  leading  companies 
expect  all  R&D  projects  to  include  prototyping  or  other  demonstrations  to 
prove  out  the  technology  before  it  is  integrated  into  a  product  for  the 
company  to  sell.  Figure  3  below  summarizes  the  general  management 
process  these  leading  companies  use  to  plan  and  execute  their  R&D 
investments. 
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Figure  3:  Commercial  Model  for  Incremental  and  Disruptive  Research  and  Development  Portfolios  Linked  to  Company 
Strategy 


Source  GAO  analysis  of  R&D  practices  of  selected  leading  companies  |  GAO-17-499 
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Leading  Companies  Use 
Corporate  Strategies  to 
Guide  Investments  in 
Portfolios  of  Incremental 
and  Disruptive 
Technologies 


Setting  Company  Strategy 


Among  the  eight  leading  companies  we  reviewed,  each  manages  R&D 
investments  that  are  underpinned  by  defined  strategies,  markets,  and 
financial  goals.  In  addition,  each  company  sets  aside  a  percentage  of 
company  revenues  to  fund  R&D.  The  company’s  strategic  direction  is  set 
by  the  Chief  Executive  Officer  (CEO),  in  coordination  with  the  company’s 
top  executives,  including  the  Chief  Technology  Officer  (CTO)  or  other 
senior  R&D  executives. 

These  corporate  strategies  balance  near-term  profitability  with  long-term 
growth  potential  and  market  expansion.  Companies  stay  competitive  by 
dividing  their  collection  of  R&D  projects,  also  known  as  their  R&D 
portfolios,  into  two  categories: 

1 .  Incremental  R&D:  lower-risk  projects  to  be  integrated  quickly  into 
near-term  products. 

2.  Disruptive  R&D:  projects  that  carry  a  higher  risk  of  failure,  but  offer 
significant  rewards  for  the  company  in  the  long-term.  These 
investments  may  lead  to  non-incremental  innovations  that  become  an 
important  piece  of  their  portfolio.  In  some  cases,  these  technologies 
render  competing  products  obsolete  by  creating  new  markets  or 
displacing  existing  product  lines. 

According  to  representatives  of  leading  companies,  around  80  percent  of 
R&D  funding  is  spent  on  incremental  development,  while  the  balance  is 
spent  on  disruptive  projects.  Corporate  leadership  determines  this 
percentage  based  on  tolerance  for  risk  and  the  company’s  financial 
standing. 

In  addition  to  these  two  portfolio  types,  leading  companies  provide 
scientists  with  the  flexibility  to  work  on  lower-cost  exploratory  projects. 
Such  work  is  conducted  by  a  few  scientists  or  researchers  and  is  not  part 
of  the  annual  process  for  approving  projects  within  each  portfolio. 
However,  the  work  derived  from  these  efforts  could  eventually  become 
part  of  incremental  or  disruptive  R&D  portfolios. 

In  determining  an  appropriate  balance  between  incremental  and 
disruptive  R&D  investments,  company  leaders  consider  long-term 
scenarios  based  on  current  trends  and  technologies  in  the  market.  For 
example,  R&D  leaders  at  Siemens  reported  that  they  conduct  an  annual 
“Innovation  Review”  of  the  company’s  entire  technology  development 
portfolio  for  the  purpose  of  informing  top  leadership’s  strategic  decisions. 
These  reviews  evaluate  Siemens’  technological  competitiveness, 
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strategic  resource  allocation,  and  long-term  corporate  strategy.  During 
this  review,  Siemens  asks  a  number  of  questions,  including  the  following: 

•  Are  the  overall  resource  allocations  for  R&D  investments  appropriate? 

•  Is  the  business  unit’s  technology  position  competitive,  and  will 
planned  investments  safeguard  the  business  unit’s  technological 
competitiveness? 

•  Is  there  a  convincing  long-term  strategy  for  how  to  translate  these 
investments  into  sustainable  business  success? 

•  Is  there  an  adequate  strategy  for  translating  new  technologies  into 
winning  offerings? 

Siemens  executives  reported  that  they  understand  that  these  factors 
directly  impact  their  ability  to  grow  and  profit  as  a  company,  which  is  why 
the  desire  to  remain  technologically  competitive  drives  corporate  strategy 
decisions. 

Siemens  also  creates  forecasting  tools  called  “Pictures  of  the  Future”  that 
provide  graphical  representations  of  how  future  technologies  could  be 
used  by  customers  7  to  15  years  in  the  future.  Figure  4  provides  an 
example  of  the  elements  that  Siemens  includes  in  a  Picture  of  the  Future. 
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Figure  4:  Siemens  Informs  Its  Research  and  Development  Strategy  by  Projecting  Pictures  of  the  Future 
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Source:  GAO  analysis  of  Siemens  AG  data  |  GAO-17-499 

According  to  Siemens,  it  uses  Pictures  of  the  Future  to 


•  assess  societal,  technological,  and  other  trends  to  guide  visionary 
concepts  for  potential  new  markets  and  customer  needs; 

•  consider  existing  product  lines,  technologies,  and  customer  needs; 
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•  analyze  the  opportunities  and  risks  for  the  company’s  core  business; 
and 

•  identify  what  is  required  to  allow  the  company  to  act  upon  potential 
future  scenarios. 

Most  importantly,  company  representatives  stated  that  these  pictures  help 
develop  consensus  within  Siemens  regarding  the  technologies  the 
company  needs  to  develop  to  drive  innovation  and  remain  a  market 
leader. 

After  leading  companies  settle  upon  their  corporate  strategies  for  R&D 
investment,  different  units  are  charged  with  sponsoring — approving  and 
funding — incremental  and  disruptive  R&D  projects.  Incremental  R&D 
projects  are  typically  sponsored  by  business  units,  who  are  also 
responsible  for  product  development.  Disruptive  R&D  is  often  sponsored 
by  a  corporate  research  organization,  which  makes  project  investment 
decisions  independently  from  the  business  units.  Figure  5  shows  this 
division  of  R&D  that  we  observed  in  the  private  sector. 


Figure  5:  Commercial  Model  Protects  Investments  for  Incremental  and  Disruptive 
Research  and  Development  (R&D)  Portfolios  Linked  to  Company  Strategy 


Strategic  direction  for  technology  development 

Leadership  defines  the  vision  for  the  company,  sets 
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on  major  projects 


strategic  plan  new  or  existing 

products 


Source  GAO  analysis  of  leading  company  R&D  pracfices  |  GAO-  I7-4S9 
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Leading  Companies  Align 
Incremental  Technology 
Investments  with  Product 
Development  Goals 


Selected  leading  companies  we  reviewed  align  plans  for  developing  new 
incremental  technologies  with  plans  for  developing  future  products,  which 
companies  sometimes  refer  to  as  roadmapping.  Individual  business  units 
are  responsible  for  product  sales  in  a  company  and  have  their  own 
executive  management  teams  that  are  charged  with  generating  profits 
from  their  product  lines.  Business  units  sponsor  incremental  development 
projects  intended  to  yield  technologies  to  meet  identified  customer  needs. 
Incremental  R&D  generally  adds  new  capabilities  to  current  products  or 
next  generation  versions  of  existing  products;  therefore  companies  expect 
these  to  have  lower  risk  of  failure.  Depending  on  the  industry,  business 
units  generally  do  not  look  beyond  a  5-year  timeframe  when  making 
decisions  about  these  new  technologies  due  to  the  unpredictable  nature 
of  the  markets  in  which  they  operate. 

These  leading  companies  document  planned  future  products  in  product 
roadmaps,  while  the  technologies  that  are  to  be  integrated  into  those 
products  are  documented  in  technology  roadmaps.  By  aligning  these 
plans,  business  units  can  better  identify  and  prioritize  technology 
development  investments.  To  develop  these  plans,  companies  solicit 
ideas  and  information  from  people  across  the  organization  to  determine 
the  composition  of  incremental  R&D  portfolios.  Ultimately,  however, 
technology  development  decisions  come  down  to  the  management’s 
qualitative  judgements  regarding  the  merits  of  individual  R&D  projects,  as 
well  as  on  quantitative  metrics  like  potential  return  on  investment.  Once  a 
project  is  approved,  it  may  be  immediately  funded  and  executed. 

The  process  and  the  number  of  people  involved  in  these  R&D  investment 
decisions  vary  depending  on  the  company.  In  general,  these  leading 
companies  solicit  input  from 

•  top  leadership  responsible  for  setting  the  company’s  overall  strategy 
and  funding,  including  the  CEO,  CTO,  and  other  corporate  leaders; 

•  representatives  from  business  units  responsible  for  getting  relevant 
products  to  market;  and 

•  scientists  and  technologists  who  plan  for  future  technology 
development  and  identify  when  technologies  are  ready  for  integration 
into  products. 

Technology  and  product  development  teams  at  Honeywell  Aerospace,  for 
example,  complete  an  annual  roadmapping  process  to  align  incremental 
technology  development  activities  with  the  company’s  product  plans.  As 
part  of  this  process, 
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•  business  units  identify  customer  needs; 

•  marketing  and  product  management  staff  review  market  trend 
information,  and  determine  future  products  and  when  they  must  be 
completed;  and 

•  Honeywell’s  corporate  research  organization  reviews  external 
technology  trends  and  creates  technology  roadmaps. 

Honeywell  Aerospace’s  roadmapping  process  is  illustrated  in  figure  6. 


Figure  6:  Honeywell  Aerospace  Integrates  Technology  and  Product  Roadmaps 


Source:  GAO  analysis  of  Honeywell  Aerospace  s  process.  |  GAO- 17-499 


Honeywell’s  roadmapping  process  focuses  on  needs  and  trends 
projected  for  the  next  10  years,  with  less  emphasis  on  the  more  distant 
future,  due  to  the  difficulty  in  making  reliable  predictions  that  far  in 
advance.  In  general,  the  leading  companies  we  met  with  consider  this 
timeframe  to  be  realistic  and  manageable  for  planning  incremental  R&D 
investments.  To  assist  with  management  decisions  regarding  what 
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incremental  R&D  projects  to  start  or  change,  Honeywell  considers 
potential  revenues  in  the  next  5  years.  However,  certain  technologies, 
such  as  those  associated  with  jet  propulsion  engines,  require  longer-term 
plans  because  technology  development  takes  many  years  to  complete. 
Although  the  company’s  technology  development  plans  do  not  extend 
beyond  the  next  10  years,  company  officials  reported  they  do  review 
industry  trends  potentially  leading  to  new  developments  further  in  the 
future  and  develop  concept  ideas  for  new  technologies  based  on  these 
trends. 

Honeywell  obtains  input  from  a  variety  of  sources  to  ensure  technologies 
will  be  feasible  for  integration  into  future  products  and  relevant  to 
customer  needs.  Technology  development  plans,  or  roadmaps,  are 
periodically  revised  based  on  changes  in  customer  needs,  prototyping 
results  not  meeting  expectations,  and  other  changes  in  circumstances 
requiring  additional  consideration.  According  to  company  representatives, 
Honeywell  stays  in  close  communication  with  customers  to  ensure  the 
company’s  understanding  of  market  needs  remains  accurate,  which  helps 
them  avoid  wasting  time  and  money  on  projects  that  have  lost  relevancy. 


Leading  Companies  Make 
Decisions  to  Fund 
Disruptive  R&D 
Independently  from 
Product  Development 
Plans 


Selected  leading  companies  we  reviewed  also  ensure  that  a  portion  of 
their  R&D  is  independently  focused  on  futuristic  concepts,  which  are 
intended  to  keep  the  companies  competitive  in  the  long  term.  Disruptive 
R&D  includes  significant  technology  development  efforts  addressing  the 
anticipated  customer  needs  of  the  future,  potentially  leading  to  products 
that  render  the  competition’s  products  irrelevant  in  the  marketplace. 

The  disruptive  R&D  portfolio  is  initiated  separately  from  incremental 
portfolios  and  often  managed  by  a  corporate  research  organization. 
Corporate  research  looks  for  solutions  that  provide  customers  with 
capabilities  they  may  not  realize  they  need  or  want.  By  separately 
organizing  disruptive  R&D  from  incremental,  companies  are  able  to 
protect  funds  from  near-term-focused  business  unit  managers.  Generally 
speaking,  companies  in  our  review  ensure  that  disruptive  R&D  is  planned 
and  executed  by  management  not  averse  to  taking  risks  when  significant 
long-term  rewards  are  possible.  Due  to  their  near-term  focus,  business 
unit  managers  usually  begin  having  significant  influence  over  disruptive 
technologies  only  after  they  have  been  demonstrated  and  are  ready  to 
begin  transitioning  into  products.  Allowing  exploratory  and  disruptive 
technology  development  to  occur  without  requiring  product  development 
approval  helps  prevent  the  company’s  products  from  becoming  obsolete 
and  gives  potential  to  capture  new  markets. 
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Companies  Challenge  Staff  to 
Develop  Ideas  for  Disruptive 
Innovation 


These  companies  use  various  approaches  to  leverage  the  ideas  of  their 
own  staff  and  external  partners  to  innovate  for  futuristic  technologies  that 
look  beyond  product  roadmaps.  These  approaches  include 

•  challenging  R&D  staff  to  come  up  with  feasible  ideas  to  create 
disruptive  technologies  leading  to  entirely  new  product  lines,  and 

•  investing  in  external  startup  companies  or  leveraging  externally 
developed  technologies. 

Companies  seek  to  ensure  longer-term  competiveness  by  challenging 
their  R&D  staff  to  make  scientific  advances  that  will  make  existing 
products  irrelevant  in  the  future.  IBM,  for  example,  issues  grand 
challenges  to  R&D  staff  to  develop  these  kinds  of  technologies.  Company 
representatives  stated  that  about  15  to  20  percent  of  IBM’s  R&D  funding 
goes  toward  development  of  disruptive  new  technologies  not  aligned  with 
known  customer  needs.  While  all  of  IBM’s  disruptive  technology 
development  is  executed  by  its  corporate  research  organization,  IBM 
business  units  provide  funding  to  sponsor  these  efforts.  To  generate 
ideas  for  disruptive  R&D  projects,  IBM  management  issues  “Grand 
Challenges”  asking  for  project  proposals.  Figure  7  below  provides 
information  on  how  IBM’s  corporate  R&D  organization  independently 
developed  the  IBM  Watson  supercomputer  as  a  result  of  one  such  Grand 
Challenge. 
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Figure  7:  IBM  Creates  Disruptive  Technologies  through  Grand  Challenges 


Pictured  here  is  IBM's  Watson  Supercomputer,  which  was 
developed  through  a  disruptive  R&D  project.  In  2007,  IBM 
management  issued  a  Grand  Challenge  to  its  corporate 
research  staff  to  make  a  computer  that  could  answer 
spoken  “Jeopardy!"  questions  and  was  capable  of 
defeating  one  of  the  show’s  champions.  While  IBM  had 
not  identified  any  external  demand  for  this  sort  of 
technology,  the  company  forecasted  that  it  could  adapt 
such  a  technology  to  address  future  customer  needs. 

After  four  years  of  development,  in  2011,  Watson  won  a 
televised  “Jeopardy!"  exhibition  match.  According  to  IBM, 
today,  Watson  is  a  central  part  of  IBM’s  Cognitive 
Solutions  product  line,  which  includes  systems  that  can 
see  and  analyze  massive  amounts  of  data.  For  example, 
Watson  provides  data  analytics  to  support  mining  explora¬ 
tion,  automotive  safety,  energy,  wealth  management,  and 
medical  treatment. 


Source  GAO  analysis  (data);  ©2012  International  Business  Machines  Corporation  (image)  |  GAO-17-499 


Amazon  also  provides  a  number  of  avenues  for  individual  staff  members 
to  submit  innovative  R&D  project  proposals  that  are  not  necessarily  tied 
to  defined  customer  needs  or  market  trends.  For  example,  company 
representatives  explained  that  Amazon  holds  week-long  innovation 
forums,  where  R&D  staff  collaborates  to  develop  new  project  ideas.  R&D 
staff  then  vote  on  the  best  ideas,  which  are  submitted  to  top  company 
leadership  for  approval.  Individual  scientists  or  technicians  can  also 
propose  new  projects  or  ideas  directly  to  their  supervisors,  who  may  help 
them  develop  formal  proposals. 

GM  initiates  disruptive  projects  through  their  corporate  research 
organization  by  funding  “Internal  Startup”  projects.  These  are  disruptive 
technology  development  projects  initiated  solely  at  the  discretion  of  GM’s 
CTO  and  the  director  of  GM’s  R&D  labs.  Aside  from  the  project  teams, 
these  are  the  only  GM  employees  that  know  the  details  of  these  projects. 
This  level  of  confidentiality  allows  GM  to  take  risks.  GM  expects  these 
projects  to 

•  have  features  that  develop  phenomenal  value  to  customers, 

•  make  existing  technology  obsolete,  and 
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Companies  Maximize  R&D 
Value  through  External 
Partnerships 


•  make  the  competition  irrelevant. 

Any  R&D  staff  member  can  propose  this  kind  of  project,  although  its 
potential  value  to  the  company  must  be  defined  in  the  proposal.  GM  also 
provides  these  projects  with  more  funding  than  typical  projects  so  that 
R&D  work  can  progress  about  three  times  faster  than  usual.  As  these 
projects  are  generally  high  in  technical  risk,  GM  officials  stated  that  they 
have  about  a  50  percent  success  rate,  which  they  considered  acceptable 
given  the  value  associated  with  successful  projects. 

Leading  companies  also  encourage  their  scientists  to  explore  and  initiate 
low-cost  research  projects,  either  as  unfunded  side-projects  or  using 
limited  resources  following  approval  from  a  supervisor.  Providing  this 
flexibility  allows  scientists  to  be  creative,  while  also  affording  them  access 
to  the  company’s  laboratory  or  other  resources.  These  less  intensive  and 
inexpensive  projects  are  not  required  to  be  approved  by  senior 
management  to  avoid  unnecessary  administrative  burden  with  scientists 
during  exploratory  development.  If  scientists  and  management  deem  a 
technology  to  be  feasible,  it  will  be  referred  to  the  company’s  R&D  project 
approval  process  for  additional  funding. 

Sometimes  leading  companies  develop  technologies  that  may  be  useful 
in  products  beyond  their  preferred  markets.  In  such  instances,  these 
companies  seek  to  maximize  the  value  of  their  R&D  investments  through 
external  partnerships.  This  may  occur  through  investments  in  startup 
companies  or  licensing  arrangements.  For  example,  Siemens  provides 
alternate  paths  for  innovative  technologies  to  move  into  products  outside 
of  their  own  product  lines  by  co-founding  start-up  companies  that  could 
turn  them  into  products,  thereby  allowing  Siemens  to  benefit  financially 
when  technologies  they  developed  are  used  in  other  companies’ 
products.  Siemens  also  licenses  some  of  its  technologies  to  other 
companies  so  they  can  be  used  in  their  products. 

Conversely,  sometimes  leading  companies  seek  technologies  from 
outside  firms  that  show  promise  for  use  in  the  company’s  own  products. 
The  Dow  Chemical  Company  uses  a  Corporate  Venture  Fund  to  make 
investments  in  companies  that  have  formed  to  commercialize  new 
technologies.  In  many  cases,  Dow  assumes  a  minority  position,  although 
in  some  cases,  Dow  may  elect  to  acquire  the  company  or  partner  without 
financial  investment  to  mutually  develop  technologies.  Dow  scientists  may 
work  alongside  R&D  staff  from  these  companies  or  Dow  may  just  choose 
to  be  an  investor.  Dow  employs  technology  scouts  around  the  world 
responsible  for  finding  opportunities  to  bring  innovative  technologies  into 
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Dow.  These  scouts  are  responsible  for  learning  everything  they  can  about 
a  company  before  Dow  proceeds  with  the  investment  or  partnership. 


R&D  Project  Teams  at 
Leading  Companies 
Collaborate  Closely  with 
Stakeholders  to  Produce 
Relevant  and  Feasible 
Technologies 


At  the  selected  leading  companies  we  reviewed,  once  an  R&D  project  is 
initiated,  the  research  team  in  charge  of  the  project  actively  collaborates 
with  stakeholders  outside  the  R&D  office  to  help  assist  and  inform  project 
execution  efforts.  These  stakeholders  typically  include 

•  product  development  staff  and  engineers  who  understand  technical 
requirements  for  technologies  to  eventually  transition  beyond  R&D, 

•  marketing  staff  familiar  with  how  products  might  fit  into  the  outside 
market, 

•  business  unit  staff  who  interface  directly  with  customers,  and 

•  potential  users  of  the  technology. 

The  level  and  timing  of  stakeholder  involvement  can  vary  based  on  the 
type  of  project.  Stakeholders,  such  as  product  development  staff,  typically 
become  involved  early  in  incremental  R&D  projects.  On  the  other  hand, 
those  same  stakeholders  might  not  get  involved  in  disruptive  projects  until 
later  phases  of  development.  Collaboration  between  stakeholders 
continues  even  after  the  technology  development  effort  concludes  and  a 
business  unit  begins  product  development.  R&D  staff  may  continue  to 
assist  product  development  efforts  after  technology  development  is 
completed  as  products  are  customized  for  different  types  of  customers. 
For  example,  figure  8  describes  how  a  cross-functional  team  at  Dow 
Chemical  collaborated  to  develop  a  new  polymer. 
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Figure  8:  Collaboration  Was  Key  to  the  Dow  Chemical  Company’s  Development  of  Olefin-Block  Co-polymers 


The  Dow  Chemical  Company’s  Olefin  Block  Co-polymers 
(OBC)  are  used  to  make  a  wide  variety  of  flexible  materials, 
such  as  shoe  mid-soles,  adhesives,  and  artificial  turf.  The 
development  of  this  product  was  considered  disruptive  since  it 
replaced  a  previous  iteration  of  Dow  polymer  products.  OBC 
development  followed  a  three-stage  process. 

Validation:  Dow’s  corporate  research  organization  led  the 
initial  research  and  development  work  as  part  of  a 
cross-functional  team  that  included  product  development  staff. 

Development:  Corporate  research  staff  assisted  with  the 
project  after  primary  responsibility  transitioned  to  product 
development. 

Implementation:  Dow’s  Elastomers  business  unit  used  the 
resulting  science  to  create  an  innovative  wrap  solution  for  the 
London  2012  Olympic  Stadium. 


Source:  GAO  analysis  (data):  ©  2013  The  Dow  Chemical  Company  (image).  |  GAO-17-499 


Leading  companies  also  look  outside  the  company  when  undertaking  an 
R&D  project  to  gain  insights  from  potential  customers.  These  customers 
provide  input  and  perspectives  that  help  inform  refinements  to 
technologies.  Figure  9  details  the  process  Floneywell  Aerospace  used  to 
seek  customer  input  when  developing  its  Synthetic  Vision  System. 
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Figure  9:  Honeywell  Aerospace  Customers  Helped  It  Develop  Synthetic  Vision  for  Aircraft 


Honeywell’s  Synthetic  Vision  System  improves  flight  crews' 
situational  awareness  by  providing  a  large,  3-D  color  synthetic 
image  of  the  outside  world.  The  initial  concept  for  the  system  was 
developed  in  2002  using  input  from  Honeywell  internal  flight  test 
pilots  and  corporate  pilots.  Honeywell  Aerospace’s  corporate 
research  organization  began  collaborating  closely  with  staff 
responsible  for  transitioning  technologies  to  product  development, 
as  the  technology’s  components  were  matured  in  a  relevant 
environment.  This  collaborative  effort  ensured  a  smooth  transition 
to  product  development.  Some  corporate  research  staff  continued 
to  work  on  developing  follow-on  features  to  further  enhance  the 
technology  as  it  became  a  product  and  to  help  aide  transition. 


Source:  GAO  analysis  (data);  ©  Honeywell  International  Inc.  (image)  |  GAO-17-499 


Company  representatives  explained  that  input  from  both  internal 
stakeholders  and  potential  customers  helps  R&D  staff  to  transition 
emerging  technologies  into  product  roadmaps.  This  collaboration  also 
helps  the  R&D  project  team  obtain  the  requisite  information  and 
resources  it  needs  to  develop  a  technology  that  is  feasible  for  use  as  part 
of  future  products,  while  also  being  relevant  to  future  customer  needs  so 
it  is  accepted  in  the  marketplace. 


Leading  Companies  Tailor 
R&D  Project  Review 
Processes  to  the  Scope  of 
Investment 


At  the  selected  leading  companies  we  reviewed,  R&D  projects  are  more 
rigorously  reviewed  by  higher  levels  of  management  as  their  needs  for 
staff  and  funding  grow.  This  leads  to  a  subset  of  these  projects  continuing 
into  later  stages  of  development  while  others  are  ended.  The  Dow 
Chemical  Company,  for  example,  uses  a  stage-gate  process  to  oversee 
R&D  projects  requiring  significant  investments.  At  each  stage  of 
development  the  project’s  funding  increases  and  the  hurdles  for  moving 
forward  become  greater.  Dow  requires  only  minimal  oversight  for  low-cost 
exploratory  research.  Once  a  scientist  proves  a  technology’s  feasibility, 
the  project  enters  Dow’s  normal  processes  for  integrated  project  oversight 
and  portfolio  management  involving  all  of  the  relevant  Dow  stakeholders. 
Later  stage  development  projects  generally  have  higher  budgets  and  are 
more  closely  monitored  to  ensure  they  meet  specific  technical  criteria  and 
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time-based  milestones,  according  to  company  representatives.  While 
specific  review  processes  vary  among  leading  companies,  figure  10 
outlines  the  key  principles  that  all  these  different  processes  embody. 


Figure  10:  As  Technologies  Mature  and  Costs  Grow,  Management  Oversight  Increases 


management  management 

review  review 

Source:  GAO  analysis  of  leading  company  R&D  practices  j  GAO-17-499 


IBM  also  emphasizes  relevant  stakeholder  participation  in  reviews  for 
R&D  projects  progressing  beyond  exploratory  or  disruptive  corporate 
research.  After  projects  are  initiated,  IBM  leadership  reviews  R&D 
projects  at  least  quarterly  during  business  reviews,  although  company 
representatives  noted  they  may  be  reviewed  more  frequently  in  some 
cases  when  more  attention  is  warranted.  IBM  leadership  believes — 
according  to  representatives  we  interviewed — that  it  must  have  agile 
review  processes  that  facilitate  timely  adjustments  or,  in  certain  cases, 
terminations  to  projects. 


Leading  Companies  Prove 
Technologies  Work  before 
Integrating  Them  into 
Products 


The  selected  leading  companies  we  reviewed  consider  technology 
demonstrations,  or  prototyping,  an  inherent  part  of  R&D  and  use 
demonstrations  for  a  variety  of  purposes,  including  to 
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•  create  demand  by  convincing  stakeholders  or  customers  of  the 
potential  value  of  a  future  technology;  and 

•  obtain  feedback  from  potential  end-users  to  add  knowledge  and 
improve  technologies. 

Both  incremental  and  disruptive  R&D  projects  receive  funds  to 
demonstrate  technologies.  Figure  1 1  depicts  the  technology 
demonstration  process  of  the  leading  companies. 


Figure  1 1 :  Product  Developers  Become  Involved  Earlier  for  Incremental  Technologies  than  Disruptive  Technologies 
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Source:  GAO  analysis  of  leading  company  R&D  practices  |  GAO-17-499 


Product  development 


Early  Prototyping  Creates 
Demand  for  New  Disruptive 
R&D  Projects 


Leading  companies  demonstrate  concepts  for  new  disruptive 
technologies  to  stakeholders  to  generate  demand  so  business  units  will 
contribute  to  technology  development.  Specifically,  once  corporate 
research  organizations  believe  technology  components  are  sufficiently 
mature  for  product  development,  they  offer  demonstrations  to  show  their 
potential  value  to  stakeholders  in  the  business  units.  For  example, 
Siemens  use  prototypes  once  a  technology  reaches  the  point  that  its 
components  can  be  validated  in  a  laboratory  or  relevant  environment. 
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Siemens  representatives  explained  that  should  the  demonstration  prove 
successful,  a  business  unit  assumes  development  responsibilities. 


Feedback  from 
Demonstrations 
Knowledge  and 
Technologies 


Leading  companies  also  demonstrate  these  concepts  for  disruptive 
technologies  to  potential  customers  if  business  units  are  hesitant  to  invest 
in  further  development.  For  example,  Qualcomm  corporate  research 
representatives  told  us  they  must  sometimes  overcome  internal 
resistance  to  accepting  new  technologies  by  demonstrating  their  value  to 
both  internal  and  external  customers.  Barriers  to  transition  of  disruptive 
technologies  may  be  even  more  prevalent  when  developing  technologies 
that  could  lead  to  dropping  an  existing  product  or  feature.  In  their  role  as 
a  R&D  component,  not  a  business  unit  focused  on  product  lines, 
corporate  research  works  with  both  internal  and  external  customers 
without  committing  to  future  products  but  with  a  clear  pathway  to  adoption 
if  successful.  Qualcomm  representatives  explained  that  if  these  early 
demonstrations  and  advocacy  for  disruptive  technologies  proves 
successful,  then  customers  may  ask  product  developers  to  use  them  in 
future  products. 

Once  a  concept  is  proven,  leading  companies  use  technology 
Add  demonstrations  to  inform  developers  how  a  technology  needs  to  be 

Improve  improved.  Company  representatives  explained  that  rapidly  developing 

and  demonstrating  a  series  of  iterations  of  a  new  technology  provides 
early  opportunities  for  improving  the  technology,  rather  than  taking  longer 
amounts  of  time  to  develop  technologies  without  testing  them.  Figure  12 
depicts  the  iterative  technology  development  and  demonstration  process 
that  leading  companies  use. 
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Figure  12:  Companies  Develop  Technologies  Iteratively  Rather  Than  in  a  Single  Step 
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Source:  GAO  analysis  of  leading  company  R&D  practices  |  GAO-17-499 
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When  Amazon  pursues  a  product  or  technology,  the  company  already 
has  an  idea  of  its  customers’  needs,  but  does  not  consider  this 
information  to  be  complete  in  the  absence  of  user  feedback  from 
prototype  demonstrations.  To  ensure  technologies  address  these  needs, 
Amazon  representatives  stated  that  the  company  builds  iterations  of 
prototypes  during  technology  development  in  a  facility  specifically 
designed  fordoing  so  quickly  and  at  minimal  cost.  Figure  13  illustrates 
how  these  early  prototypes  are  generally  used  by  Amazon  employees  in 
real  world  settings  to  help  inform  technology  development. 
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Figure  13:  Amazon  Staff  Inform  Technology  Development  by  Testing  Prototypes  in  Their  Homes 


Amazon  development  staff  tested  prototypes  of  the  company’s 
internet-enabled  “Echo”  device — a  hands-free  speaker 
controlled  by  voice  that  can  interface  with  and  manage  home 
devices,  play  music,  and  provide  information — in  their  homes 
before  the  product  was  released  to  market.  In  conducting  this 
testing,  the  development  staff  provided  “bug  reports”  to 
document  problems  with  the  Echo  prototypes.  Amazon 
collected  this  data,  along  with  insights  on  how  these  early 
customers  used  Echo,  to  create  additional  prototypes  that 
incorporated  necessary  changes. 


Source  GAO  analysis  (data).  Amazon  com  (image)  |  GAO-17-499 


Similar  to  Amazon,  IBM’s  development  method  uses  prototyping  during 
technology  and  product  development.  IBM  developers  show  early 
iterations  of  new  technology  to  customers,  improve  the  design  based  on 
feedback,  and  then  produce  another  prototype  after  that.  This  process 
repeats  until  a  releasable  product  is  completed.  IBM  representatives 
found  that  development  models  with  longer  sequential  steps  and  a  single 
deliverable  are  less  useful  than  faster  paced  and  smaller  deliverables 
under  shorter  timeframes. 

Companies  may  also  use  external  demonstrations  of  maturing 
technologies  to  prove  they  work  in  realistic  environments.  For  example, 
Valvoline  produces  and  tests  small  volumes  of  new  motor  oil  formulas  in 
its  own  laboratory  or  at  a  few  select  customers’  facilities.  To  obtain 
customer  insights  from  prototyping,  Valvoline 

•  develops  data  and  uses  cameras  on  test  engines  to  demonstrate  the 
formulas  performance, 

•  uses  customer  test  engines  to  conduct  tests  and  provide  specific  data 
to  customers  on  how  new  formulas  perform  in  their  engines;  and 
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•  provides  test  formulas  to  outside  customers  and  obtains  feedback 
through  sales  staff  and  focus  groups. 

Offering  external  customers  an  opportunity  to  test  new  technologies 
provides  important  feedback  for  companies  such  as  Valvoline.  By  taking 
steps  like  these  to  involve  outside  customers  in  prototype  testing, 
representatives  of  leading  companies  stated  they  are  able  to  identify  a 
new  technology’s  tangible  benefits,  while  also  encouraging  eventual 
customer  acceptance  of  new  products. 


DOD  Implements 
Some  Practices  Used 
by  Leading 
Companies,  but 
Policy  Limitations  and 
Culture  Hinder 
Adopting  Others 


While  some  DOD  S&T  practices  closely  mirror  those  of  the  selected 
leading  companies  we  reviewed,  DOD’s  funding  policies  and  culture  limit 
its  ability  to  adopt  other  practices  for  managing  its  S&T  investments. 

Unlike  the  companies  we  reviewed,  DOD  does  not  organize  and  fund 
incremental  and  disruptive  innovation  separately.  Nor  does  its  leadership 
provide  guidance  on  or  assess  how  these  innovation  investments  should 
be  or  are  mixed.  Instead,  S&T  officials  explained  that  DOD  labs  face 
pressures  to  prioritize  near-term  requirements  at  the  expense  of 
potentially  disruptive  technologies.  As  a  result  of  DOD  funding  policies, 
projects  are  planned  2  years  in  advance,  which  can  slow  innovation  and 
limit  lab  directors’  autonomy  to  initiate  work.  While  Congress  has  provided 
authority  that,  as  implemented,  has  enabled  the  military  department  lab 
directors  to  initiate  work  outside  of  the  normal  lengthy  process,  DOD  has 
not  fully  utilized  these  flexibilities.  Additionally,  we  found  that  divided 
responsibilities  for  technology  versus  product  development  contribute  to  a 
culture  that  does  not  encourage  collaboration  between  DOD’s  S&T  and 
acquisition  communities  and  limits  the  S&T  community’s  ability  to  conduct 
advanced  prototyping.  These  issues  are  not  insurmountable,  however,  as 
demonstrated  in  pockets  of  each  military  department.  In  recognition  of 
these  and  other  issues,  Congress  has  required  that  DOD  create  a  new 
Under  Secretary  of  Defense  for  Research  and  Engineering  charged  with 
advancing  defense  technology  and  innovation  and  establishing  policies 
on  technology  development,  prototyping,  and  experimentation,  among 
other  responsibilities,  by  February  2018. 


Some  DOD  Practices  for 
Managing  and  Executing 
S&T  Investments  Closely 
Resemble  Practices  Used 
by  Leading  Companies 


Some  of  DOD’s  practices  for  managing  and  executing  S&T  investments 
closely  resemble  those  employed  by  the  selected  leading  companies  we 
reviewed.  DOD  has  a  corporate  research  organization  for  disruptive 
innovation  and  its  leadership  defines  S&T  strategies  to  guide  investments 
which  are  consistent  with  elements  that  the  companies  we  met  with  used 
to  manage  investments  in  incremental  and  disruptive  portfolios.  DOD 
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project  oversight  is  scaled  based  on  the  scope  of  investment,  which 
aligns  closely  with  leading  company  practices. 

While  differences  exist  between  how  these  practices  are  implemented  in 
DOD  and  at  the  companies  we  reviewed,  we  found  that  the  outcomes  are 
the  same. 

•  The  Defense  Advanced  Research  Projects  Agency  (DARPA),  for 
example,  closely  resembles  the  corporate  research  organization  that 
many  leading  companies  employ  to  foster  disruptive  innovation.  In  the 
President’s  budget  submission  for  fiscal  year  2017,  DARPA  requested 
$2.9  billion,  about  23  percent  of  DOD’s  S&T  budget  request.  Similar  to 
a  company’s  corporate  research  organization,  DARPA’s  projects  are 
generally  not  tied  to  existing  DOD  weapon  systems  or  a  specific 
military  department  requirement.  Instead,  their  mission  is  to  produce 
disruptive  innovation  that  could  support  any  military  department.  The 
DARPA  Director  makes  all  funding  decisions  and  project 
prioritizations. 

•  DOD’s  market  research,  which  informs  its  S&T  strategy,  is  based  on 
near  and  far-term  adversarial  threats,  capability  needs,  and  warfighter 
requirements.  While  these  inputs  may  differ  from  the  companies  we 
reviewed,  they  are  likewise  used  to  prioritize  S&T  projects. 

•  DOD  does  not  necessarily  use  the  same  metrics  as  companies  to 
evaluate  projects,  but  its  labs  similarly  scale  the  scope  of  their  project 
reviews  based  on  the  maturity  of  the  technology  and  scope  of 
investment.  For  example,  a  lab  typically  reviews  basic  research 
projects  once  a  year,  while  officials  said  more  mature,  larger 
investments  are  reviewed  multiple  times  per  year  by  the  lab,  its 
customers,  and  military  department  leadership.  Similarly,  both  DOD 
and  the  companies  we  reviewed  assess  projects  based  on  their  cost, 
schedule,  and  technical  performance  requirements.  One  inherent 
difference  is  that  leading  companies  are  concerned  with  potential 
financial  returns  on  investment,  whereas  DOD  prioritizes  for  other 
reasons,  such  as  whether  the  technology  carries  potential  to  reduce 
risk  to  the  warfighter. 
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Military  Departments  Do 
Not  Define  in  Strategy  or 
Assess  the  Mix  of 
Investments  in 
Incremental  and  Disruptive 
Innovation 


Although  our  review  of  selected  leading  companies  found  that  they  define 
in  strategy  their  annual  mix  of  investments  in  incremental  and  disruptive 
innovation,  the  military  departments  do  not  do  this,  nor  do  they  assess 
such  a  mix.  The  office  of  the  ASD(R&E) — the  Office  of  the  Secretary  of 
Defense  (OSD)  organization  responsible  for  establishing  DOD  S&T  policy 
and  guidance — does  not  provide  guidance  to  the  military  departments  on 
the  mix  of  incremental  and  disruptive  S&T  investments.  The  military 
departments  are  responsible  for  defining  their  own  S&T  strategies, 
formulating  and  managing  budgets,  and  developing  technologies.  Their 
S&T  strategies,  however,  do  not  define  the  mix  of  incremental  and 
disruptive  investments  each  department  should  make  annually.  Instead, 
DOD  S&T  investments  are  organized  and  funded  based  on  budget 
activities  (BA)  that  reflect  stages  of  technology  maturity.6  DOD  uses  this 
approach,  in  part,  because  the  Financial  Management  Regulation  dictates 
how  R&D  activities  are  identified  for  the  purposes  of  budgeting.  We  found 
that  one  limitation  of  funding  under  BAs  is  a  lack  of  visibility  into  whether 
individual  projects  that  labs  and  research  centers  invest  in  are  geared 
toward  disruptive  or  incremental  innovation.  The  Financial  Management 
Regulation,  however,  does  not  preclude  DOD  from  developing  investment 
targets  for  both  incremental  and  disruptive  R&D. 


Military  department  lab  and  center  officials  we  interviewed,  however, 
identified  certain  projects  they  were  working  on  that  could  lead  to 
disruptive  technologies.  Officials  from  these  labs  and  centers 
acknowledged  that  they  struggle  to  determine  the  right  balance  between 
disruptive  and  incremental  innovation  projects.  They  expressed  concern 
that  military  department  leadership  responsible  for  setting  requirements 
for  and  approving  S&T  spending,  at  times,  are  more  focused  on  near- 
term,  less  risky,  more  incremental  types  of  innovation  investments  at  the 
expense  of  long-term,  disruptive  innovation. 


The  Navy  is  one  military  department  that  has  taken  steps  to  ensure 
funding  for  some  investments  in  disruptive  innovation.  The  Navy 
organizes  S&T  investments  around  “strategic  buckets”  to  ensure  it 
maintains  investments  in  both  near-  and  long-term  projects  and  protects 
funding  for  potentially  disruptive  projects.  The  distribution  of  resources  is 
determined  by  senior  leadership  based  on  the  Navy’s  S&T  strategy.  The 
strategy  maps  out  roughly  the  minimum  percentage  of  funding  that  the 


6See  Department  of  Defense  Financial  Management  Regulation  (DOD  FMR),  DOD 
7000.1 4-R  (Dec.  2016). 
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Navy  plans  to  request  for  high-priority,  disruptive  projects  within  its  S&T 
portfolio,  as  reflected  in  figure  14  below.  In  fiscal  year  2017,  the  Navy 
plans  to  invest  more  than  $313  million  for  Leap  Ahead  Innovations  which 
are  intended  to  be  disruptive  technologies  and  deliver  transformational 
warfighting  capabilities.  These  are  in  addition  to  investments  in  other 
disruptive  technologies  that  are  categorized,  but  not  quantified,  under  its 
other  strategic  buckets. 


Figure  14:  The  Navy  Reports  It  Ensures  Investments  in  Disruptive  Innovation  by  Using  a  “Strategic  Buckets”  Funding 
Approach 
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Note:  Discovery  and  Innovation  includes  basic  and  applied  research  (6.1  and  6.2).  Leap-Ahead 
Innovations  refer  to  the  technology  portfolio  that  defines  the  future  of  naval  warfighting.  Acquisitions 
Enablers  deliver  critical  component  technologies  to  naval  acquisition  programs.  Quick  Reaction 
responds  to  urgent  technology  needs  and  solves  problems  for  warfighters.  Existing  systems  refers  to 
projects  responsive  to  immediate  needs  or  compelling  innovation  identified  by  Navy  leadership. 


In  a  June  2017  report,  we  recommended  that  DOD  take  steps — such  as 
the  Navy’s — to  help  ensure  adequate  investments  in  innovation  that  align 
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with  DOD-wide  strategy  to  overcome  the  department’s  risk-averse  culture 
and  pressures  to  focus  on  near-term  projects.7 


DOD’s  Lengthy  S&T 
Project  Planning  and 
Budgeting  Cycle  Can  Slow 
Innovation  While  Existing 
Flexibilities  to  Expedite 
Have  Been  Underused 

•  Planning:  DOD  leadership,  in  guidance  and  planning  documents, 
identifies  strategic  priorities,  weapon  system  requirements,  and 
adversarial  threats.  Collectively,  these  serve  as  DOD’s  broad 
requirements  for  technology  development. 

•  Programming:  S&T  organizations  give  consideration  to  those 
requirements  and  propose  technology  development  projects  to 
address  them.  Proposed  projects  and  associated  costs  are 
documented  in  Program  Objective  Memorandums  (POM).  Each 
organization  is  tasked  with  determining  which  projects  to  propose  in 
the  POM,  while  maintaining  balance  across  their  portfolios  of 
investment,  as  well  as  maintaining  an  appropriate  mix  of  funding 
based  on  BA.  POM  documents  are  reviewed  by  senior  officials  across 
DOD — including  those  responsible  for  setting  requirements  and  the 
budget — who  also  have  a  role  in  prioritizing  S&T  investments. 

•  Budgeting:  Each  S&T  organization’s  POM  is  used  to  formulate  their 
respective  military  department’s  Budget  Estimate  Submission  (BES), 
which  outlines  the  total  funding  needed,  including  how  much  it  will 
need  by  budget  activity.  After  the  President’s  budget  is  submitted, 
Congress  enacts  an  appropriation.  Once  funds  are  appropriated,  each 
S&T  organization  is  provided  funding  for  the  projects  approved  in  the 
POM  and  BES. 

•  Execution:  S&T  organizations  carry  out  funded  projects. 


In  comparison  to  the  practice  at  selected  leading  companies  we  reviewed, 
which  annually  align  their  investments  to  product  goals,  DOD’s  process 
for  prioritizing  and  funding  projects  takes  longer — almost  2  years  to 
complete — which  we  found  can  slow  innovation.  Like  every  other  good 
and  service  DOD  acquires,  all  S&T  investments  must  follow  DOD’s 
planning  and  budgeting  policy.8  This  policy  is  underpinned  by  DOD’s 
Planning,  Programming,  Budgeting,  and  Execution  (PPBE)  process.  The 
PPBE  process  for  S&T  investments  includes  the  following  stages: 


7GAO,  Weapon  Systems:  Prototyping  Has  Benefited  Acquisition  Programs,  but  More  Can 
Be  Done  to  Support  Innovation  Initiatives ,  GAO-17-309,  (Washington,  D.C.,  Jun.  27, 
2017). 

department  of  Defense  Directive  7045.14,  “The  Planning,  Programming,  Budgeting,  and 
Execution  (PPBE)  Process”  (Jan.  25,  2013). 
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Figure  15  illustrates  the  notional  time  frames  for  DOD’s  PPBE  process. 


Figure  15:  Notional  Timeframes  for  the  Department  of  Defense’s  (DOD)  Planning,  Programming,  Budgeting,  and  Execution 
Process 
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Source  GAO  analysis  of  DOD  documents  |  GAO-17-499 


In  total,  it  can  take  almost  2  years  from  the  time  a  project  is  proposed  in 
the  POM  to  the  time  it  is  funded.  In  contrast,  the  companies  we  reviewed 
reported  that  they  planned  projects  in  the  same  year  they  were  executed, 
which  helped  them  quickly  respond  to  leaps  in  technology  development. 
S&T  officials  we  met  with  stated  that  the  2-year  project  planning  process 
reduces  their  ability  to  be  as  nimble  as  the  companies  with  whom  we  met, 
For  example,  if  an  unexpected  technology  breakthrough  is  identified 
through  6.1  or  6.2  research,  the  labs  may  have  to  wait  up  to  2  years 
before  they  may  begin  work  on  a  follow-on  project.  DOD  S&T  executives 
expressed  the  need  for  greater  flexibility  with  initiating  new  projects 
because  the  pace  of  technology  development  can  be  rapid  and  planning 
for  S&T  spending  2  years  in  advance  can  hinder  innovation.  They  stated, 
however,  that  the  PPBE  process  provides  Congress  with  the  information 
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it  needs  to  maintain  oversight  and  ensure  DOD  meets  its  fiduciary 
responsibilities  to  the  taxpayer. 

We  found  that  laboratory  and  research  center  directors  in  the  military 
departments  have  less  authority  to  initiate  S&T  work  that  is  not  directly 
linked  to  defined  near-  or  far-term  capability  needs  as  compared  to  the 
leading  companies  we  reviewed.  While  the  Director  of  DARPA  approves 
every  project  the  agency  undertakes  and  is  not  beholden  to  address 
defined  requirements,  the  military  departments’  labs  and  centers  do  not 
control  all  of  the  S&T-related  R&D  work  they  perform  annually.  These 
labs  and  centers  regularly  undertake  work  on  behalf  of  acquisition 
community  customers,  such  as  a  major  defense  acquisition  program,  who 
provide  funding  in  support  of  the  project.  This  work  comes  in  addition  to 
“direct  funded”  projects  approved  and  funded  to  the  lab  or  center  through 
PPBE  that  are  intended  to  address  S&T  requirements  outlined  in  strategy. 
As  a  result,  both  the  direct  funded  and  customer-funded  projects  compete 
for  lab  resources,  such  as  staff,  and  must  be  balanced.  In  fiscal  year 
2015,  for  example,  direct  funded  projects  accounted  for  19  percent  of 
Naval  Research  Laboratory’s  (NRL)  $1.2  billion  of  funding.  The  other  81 
percent  was  customer-funded  work  from  Navy,  other  DOD,  or  other 
governmental  sources.  Despite  having  direct  funding,  the  POM  review 
and  approval  process  may  constrain  which  projects  are  ultimately  funded. 
For  example,  we  found  that  the  projects  a  lab  or  center  proposes  in  its 
POM  submission  may  be  reviewed  by  as  many  as  four  different 
organizations  before  it  is  submitted  to  OSD.  It  is  during  this  review 
process  that  lab  officials  explained  that  the  culture  within  DOD  is,  at 
times,  to  focus  on  near-term,  customer-driven  projects,  at  the  expense  of 
far-term  disruptive  projects. 

Regardless  of  the  source  of  the  funding,  a  senior  ASD(R&E)  official 
explained  that  S&T  investments  are  intended  to  address  some  defined 
capability  need.  This  means  that  the  military  department’s  disruptive 
technology  projects  are  roadmapped  to  requirements,  which  differs  from 
the  practice  at  the  leading  companies  we  reviewed.  This  may  limit  the 
labs’  ability  to  address  undefined  customer  needs  through  other 
potentially  disruptive  technologies. 

Section  219  of  the  Duncan  Hunter  National  Defense  Authorization  Act  for 
Fiscal  Year  2009,  as  implemented,  has  provided  defense  lab  directors 
with  some  limited  flexibility  to  initiate  S&T  projects,  including  those  that 
are  not  roadmapped  to  defined  requirements,  outside  of  the  normal  2- 
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year  planning  process.9  Specifically,  as  amended,  the  law  directs  the 
Secretary  of  Defense,  in  consultation  with  the  Secretaries  of  the  military 
departments,  to  establish  mechanisms  under  which  the  director  of  a 
defense  laboratory  may  use  an  amount  of  funds  equal  to  a  certain 
percentage  of  all  funds  available  to  the  laboratory  for  the  following 
purposes: 

•  innovative  basic  and  applied  research  that  is  conducted  at  the 
defense  laboratory  and  supports  military  missions; 

•  developing  programs  that  support  the  transition  of  technologies 
developed  by  the  defense  laboratory  into  operational  use; 

•  workforce  development  activities  that  improve  the  capacity  of  the 
defense  laboratory  to  recruit  and  retain  personnel  with  needed 
scientific  and  engineering  expertise;  and 

•  revitalization,  recapitalization,  or  minor  military  construction 
of  laboratory  infrastructure. 

While  this  authority  directed  the  creation  of  a  mechanism  that  may 
provide  lab  directors  with  the  means  to  fund  projects  they  consider  to  be  a 
priority,  the  military  departments  have  not  maximized  their  use  of  these 
authorities.  Until  the  passage  of  the  National  Defense  Authorization  Act 
for  Fiscal  Year  2017,  the  director  of  a  defense  lab  could  use  funds  equal 
to  not  more  than  3  percent  of  all  funds  available  to  the  defense  laboratory 
for  S&T  activities.  Each  of  the  services  has  unique  strategies  for 
executing  section  219  authorities  but  DOD  reported  that  the  full  3  percent 
of  funds  available  to  the  labs  has  not  been  used.  DOD  officials  told  us 
that  the  full  3  percent  available  to  each  defense  laboratory  has  not  been 
used  for  a  number  of  reasons,  including  due  to  competing  S&T  funding 
priorities.  Additionally,  DOD  officials  indicated  that  labs  had  concerns 
about  charging  customers  a  fee  to  fund  such  S&T  activities,  which  was  a 
factor  in  different  amounts  of  funds  available  for  section  219  purposes,  as 
shown  in  figure  16. 


9Pub.  L.  No.  1 1 0-41 7  §  21 9  (2008),  as  amended  at  1 0  U.S.C.  §  2358  note. 
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Figure  16:  Fiscal  Year  2015  Section  219  Funds  by  Each  Military  Department  (dollars 
in  millions) 
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During  our  review,  Congress  amended  the  Section  219  authority  in  the 
National  Defense  Authorization  Act  for  Fiscal  Year  2017  to  permit  lab 
directors  to  use  an  amount  of  funds  not  less  than  2  percent  but  not  more 
than  4  percent  of  all  funding  available  to  the  lab.10  As  a  result  of  the 
change,  the  military  departments  can  increase  the  amount  of  section  219 
funds  that  the  labs  may  obtain.  The  extent  to  which  they  choose  to  fund 
new  projects  could  help  them  to  initiate  projects,  including  those  which 
may  be  “off-roadmap,”  faster  than  through  the  PPBE  process. 


10The  National  Defense  Authorization  Act  for  Fiscal  Year  2017,  further  authorized  defense 
lab  directors  to  charge  customer  activities  a  fixed  percentage  fee,  in  addition  to  normal 
costs  of  performance,  of  up  to  four  percent ,  in  order  to  obtain  funds  to  carry  out  activities 
authorized  under  Sec.  219  authority.  Pub.  L.  No.  1 1 4-328  §  21 2  (201 6). 
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Cultural  Barriers  That  Limit 
Collaboration  between 
DOD’s  S&T  and 
Acquisition  Communities 
Are  Not  Insurmountable 


While  the  selected  leading  companies  we  reviewed  ensure  close 
collaboration  between  stakeholders  in  technology  and  product 
development,  cultural  barriers  have  limited  such  collaboration  within 
DOD.  DOD’s  funding  policies  reflect  cultural  barriers  to  collaboration 
between  the  S&T  community  and  its  product  development  stakeholders. 
Under  DOD’s  funding  model,  the  labs  are  responsible  for  technology 
development  associated  with  BAs  6.1  through  6.3,  while  the  stakeholders 
in  the  acquisition  community  are  traditionally  responsible  for  product 
development,  which  begins  with  prototype  activities  under  BA  6.4. 11 
Although  we  found  that  S&T  projects  funded  by  the  acquisition  community 
obtain  collaborative  input  from  those  same  eventual  customers,  this  was 
not  the  case  for  direct-funded  projects — those  initiated  by  the  lab.  Lab 
officials  explained  that  for  direct-funded  projects,  they  may  consult  with 
potential  customers  in  the  acquisition  community  to  gauge  interest  before 
starting  a  project  or  to  present  results  after  it  is  complete,  but  those 
customers  are  not  part  of  the  development  team.  This  approach  is,  in 
part,  attributable  to  these  organizations  being  separate — both  in  mission 
and  in  the  type  of  funding  they  receive.  For  example,  unlike  the 
companies  we  reviewed,  S&T  officials  explained  that  they  do  not 
transition  scientists  and  engineers  along  with  the  technologies  they 
developed  to  the  acquisition  programs.  The  companies  we  reviewed 
reported  that  they  set  the  expectation  that  both  technology  and  product 
development  staff  work  together.  DOD,  however,  has  not  established  a 
formal  policy  on  how  these  two  communities  should  collaborate  on 
projects  to  overcome  these  cultural  barriers. 


DOD  has  processes  to  help  its  research  labs  and  centers  collaborate  on 
S&T  work,  but  these  processes  do  not  emphasize  collaboration  between 
the  S&T  and  acquisition  communities.  In  2014,  ASD(R&E)  revitalized  its 
“Reliance  21”  framework — a  joint  planning  and  coordination  process  that 
is  intended  to  ensure  DOD’s  S&T  community  provides  solutions  and 
advice  to  the  departments’  senior-level  decision  makers,  warfighter, 
Congress,  and  other  stakeholders.  This  is  to  be  accomplished,  in  part, 
through  groups  of  technical  experts  organized  around  17  technical  areas 
referred  to  as  Communities  of  Interest  (COI).  Originally  formed  in  2009, 
COIs  provide  DOD  with  a  mechanism  for  experts  in  technical  areas,  such 
as  cyber  or  space,  to  coordinate  and  communicate  what  S&T-related 
R&D  each  military  department  is  working  on  and  identify  areas  for 


11  BA  1  through  3  includes  S&T  R&D  for  basic  research,  applied  research,  and  advanced 
technology  development. 


Page  35 


GAO-17-499  Innovation  Best  Practices 


collaboration.  Each  COI  is  supposed  to  conduct  a  portfolio  review  every  2 
to  3  years — depending  on  the  technical  area — to  assess  those  gaps  and 
their  impacts  and  to  make  recommendations  to  the  S&T  Executive 
Committee.  We  found  that  each  COI  has  documented  at  least  one  such 
assessment  since  2014.  According  to  an  ASD(R&E)  official,  each  COI 
also  provides  updates  to  the  S&T  Executive  Committee  during  its  annual 
S&T  strategy  meeting. 

DOD’s  S&T  executives  stated  that  there  is  a  need  to  improve  how  and 
when  the  acquisition  community  is  brought  in  to  contribute.  One  S&T 
executive  pointed  out  that  bringing  in  external  stakeholders  earlier  in  the 
process  is  a  way  to  facilitate  disruptive  innovation.  They  described  one 
instance  in  which  the  Air  Force  overcame  existing  cultural  barriers  to 
collaboration  by  funding  technology  maturation  efforts  with  both  S&T  and 
acquisition  community  support,  as  described  in  figure  17.  This  different 
approach  was  the  result  of  a  deliberate  decision  by  the  Air  Force  to  foster 
more  collaboration  between  the  S&T  community  and  other  stakeholders 
to  increase  the  odds  that  new  technologies  end  up  in  the  hands  of  the 
warfighter,  according  to  officials. 
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Figure  17:  Close  Collaboration  Led  to  Successful  Development  of  the  Air  Force’s  Adaptive  Engine  Technologies 


The  Air  Force  Life  Cycle  Management  Center's  (AFLCMC) 
Adaptive  Engine  Transition  Program  (AETP)  is  an 
example  of  close  collaboration  between  the  science  and 
technology  (S&T)  and  acquisition  communities.  Adaptive 
engine  technology  is  intended  to  deliver  advanced 
propulsion  capabilities  for  future  tactical  aircraft,  and  to 
make  these  aircraft  more  fuel  efficient.  According  to  Air 
Force  officials,  the  Air  Force  Research  Laboratory  initiated 
technology  development  efforts  using  S&T  funding.  Once 
the  technology  was  ready  for  transition  to  a  flight-weight 
prototyping  effort  needed  advanced  prototype  funding  but 
there  was  no  specific  acquisition  program  ready  to  further 
mature  and  integrate  the  engine  technologies. 

The  Air  Force,  however,  decided  to  proceed  with  the 
prototyping  effort  outside  of  a  weapons  system  acquisition  program  to  avoid  any  delays  with  maturing  the  technology.  AFLCMC 
initiated  its  AETP  prototyping  efforts  prior  to  completion  of  the  Air  Force  Research  Lab’s  which  created  about  an  18-month  overlap 
of  the  two  programs  According  to  the  Air  Force,  this  overlap  provided  an  opportunity  to  leverage  experience  from  the  Air  Force 
Research  Laboratory's  S&T  efforts  and  inform  the  prototyping  design  efforts,  enabling  more  efficient  technology  maturation. 
Additionally,  scientists  and  engineers  from  the  Air  Force  Research  Laboratory  continued  to  support  the  new  AFLCMC  acquisition 
team’s  prototyping  efforts.  An  Air  Force  Research  Laboratory  official  stated  that  although  the  program  is  a  high  priority  for  the  Air 
Force  and  had  strong  support  from  the  Office  of  the  Secretary  of  Defense,  this  close  collaboration  between  the  acquisition  and 
S&T  communities  is  a  primary  reason  the  adaptive  engine  development  effort  is  meeting  its  technical  objectives. 


Source  GAO  analysis  of  Air  Force  information  (data).  GE  Aviation  (image)  |  GAO-17-499 


Military  Department  Labs 
Traditionally  Funded 
Technology  Development 
through  Initial  Prototypes, 
but  Recent  Initiatives  Have 
Begun  to  Fund  Advanced 
Prototyping 


While  selected  leading  companies  we  reviewed  provided  funding  for 
prototypes  during  technology  development,  DOD  has  only  recently  begun 
to  fund  advanced  prototyping  efforts  within  the  labs.  The  acquisition 
community,  as  opposed  to  the  S&T  labs,  traditionally  bears  the 
responsibility  of  maturing  technology  through  advanced  prototyping, 
which  is  in  contrast  to  how  the  companies  with  whom  we  met  operate. 
DOD’s  S&T  labs  and  centers  typically  do  not  control  the  BA  6.4  funding 
that  would  allow  them  to  conduct  such  prototyping.  We  found  that  the 
S&T  community  typically  matures  technologies  to,  at  most,  a  prototype 
that  is  close  to  final  form,  fit,  and  function  and  tested  in  a  relevant 
environment.  This  creates  strong  incentives  for  S&T  project  teams  to 
identify  technology  transition  partners  in  the  form  of  major  acquisition 
programs  early  in  development,  which  may  ultimately  restrict  disruptive 
innovation  and  push  S&T  projects  to  be  more  incremental  to  satisfy 
potential  customers’  near-term  needs.  For  example,  applied  research 
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project  proposals  at  the  Naval  Research  Lab  and  the  Army’s  Engineer 
Research  and  Development  Center  both  identify  and  consider  potential 
transition  partners  as  part  of  the  selection  criteria,  regardless  of  whether  it 
was  for  incremental  or  disruptive  technologies.  As  we  previously  stated,  it 
is  difficult  to  transition  a  technology  or  identify  a  partner  when  the 
technology  is  disruptive.  Companies  recognized  this  and  funded 
disruptive  technology  development  projects  through  demonstration  to 
help  obtain  a  customer. 

In  June  2017,  we  reported  that  DOD’s  approach  to  prototyping 
contributes,  in  part,  to  DOD’s  broad  challenges  with  transitioning 
technology  from  the  labs  into  the  hands  of  the  warfighter.12  We  further 
reported  that  prototyping  that  is  not  directly  tied  to  acquisition  programs 
can  be  seen  as  a  way  to  “test  the  waters”  because  it  does  not  require  the 
level  of  commitment  associated  with  starting  acquisitions. 

Each  military  department  has  recently  undertaken  efforts  to  fund  more 
advanced  prototypes  for  incremental  and  disruptive  technologies  in  S&T 
labs.  For  example,  since  2012,  the  Army  has  used  funding  typically 
associated  with  acquisition  programs  to  conduct  higher-fidelity  prototyping 
and  further  mature  technology  outside  of  those  programs  through  its 
Technology  Maturation  Initiative.  In  the  President’s  fiscal  year  2017 
budget  submission,  the  Army  requested  approximately  $70  million  for 
these  efforts. 

In  May  2016,  the  Air  Force  established  the  Strategic  Development 
Planning  and  Experimentation  Office,  in  part,  to  run  Air  Force 
experimentation  initiatives  to  achieve  specific  technology  development 
objectives.  Air  Force  officials  explained  that  the  intent  of  these  initiatives 
is  to  develop  more  agile  approaches  to  innovation  by  creating  a  learning 
organization  that 

•  can  rapidly  take  new  innovative  approaches, 

•  can  quickly  initiate  new  projects  and  is  not  hampered  by  the  traditional 
2-year  planning  process, 

•  is  composed  of  acquisition  and  S&T  representatives  to  promote 
collaboration,  and 


12GAO-1 7-309. 
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•  conducts  prototyping  and  demonstrations  without  a  direct  requirement 
from  an  acquisition  community  stakeholder  to  reduce  risk  and  mature 
technologies. 

In  the  President’s  fiscal  year  2017  budget  request,  the  Air  Force 
requested  $62  million  to  fund  these  experimentation  efforts.  Currently, 
two  technology  development  areas  are  addressed  through 
experimentation  initiatives  and  two  more  are  planned.  According  to  Air 
Force  officials,  these  initiatives  reflect  its  leadership’s  desire  to  embrace  a 
culture  of  encouraging  and  formulating  innovative  strategic  choices 
independent  of  major  weapon  system  acquisition  programs. 

The  Department  of  Navy  is  pursuing  similar  efforts  through  its  Rapid 
Prototyping,  Experimentation,  and  Demonstration  (RPED)  initiative. 

RPED  projects  use  prototyping  to  rapidly  develop  and  assess  new 
technologies  and  engineering  innovations  to  address  priority  naval 
warfighting  needs.  The  Navy  expects  that  RPED  projects  will  assist  in 

•  developing  new  capability  concepts, 

•  informing  and  refining  requirements, 

•  addressing  priority  needs  by  demonstrating,  and 

•  enabling  quicker  transition  of  technologies  to  naval  programs. 

The  Navy  developed  its  policy  for  RPED  projects  in  December  2016  and 
requested  $40  million  to  fund  projects  in  the  President’s  fiscal  year  2017 
budget  submission.13 

In  our  June  2017  report,  we  recommended  that  DOD  develop  a  strategy 
to  better  coordinate  and  communicate  the  goals  of  these  and  other 
prototyping  efforts  to  ensure  these  efforts  gain  traction  and  achieve 
success.14 

An  October  2013  Defense  Science  Board  report  reinforces  the  military 
departments’  focus  on  experimentation  as  an  innovation  enabler.15  The 


1  Secretary  of  the  Navy  Instruction  5000.42,  “Department  of  the  Navy  Accelerated 
Acquisition  for  the  Rapid  Development,  Demonstration  and  Fielding  of  Capability”  (Dec. 
22,2016). 

14GAO-1 7-309. 

15Defense  Science  Board,  Technology  and  Innovation  Enablers  for  Superiority  in  2030 
(Washington,  D.C.:  Oct.  2013). 
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Defense  Science  Board  found  that  DOD  cannot  continue  to  rely  on 
technological  superiority  unless  it  adopts  methods  that  allow  it  to 
anticipate,  assess,  and  gain  experience  with  new  technological 
capabilities  before  its  potential  adversaries  do. 


DOD  and  Commercial 
Firms  Operate  under 
Different  Incentives  and 
Organizational  Structures 


DOD’s  organizational  structure  and  incentives  contribute  to  why  it  does 
not  fully  implement  the  S&T  management  practices  that  the  selected 
leading  companies  we  reviewed  follow.  This  includes  DOD’s  budget 
environment,  funding  model,  and  the  manner  in  which  DOD  is  organized 
to  execute  technology  development.  As  we  have  previously  reported,  the 
critical  differences  between  the  environments  and  cultures  of  private 
companies  and  DOD  must  be  recognized  before  tangible  progress  can  be 
made  in  establishing  more  efficient  practices  in  S&T  management. 
Further,  we  concluded  that  changing  the  mechanics  of  the  processes, 
without  changing  the  environment  that  determines  incentives,  may  not 
produce  better  outcomes.16  Specifically: 


•  Companies  operate  in  an  environment  where  profitability  is  a  constant 
business  imperative.  As  such,  leading  companies  we  reviewed 
devoted  a  portion  of  their  R&D  investments  toward  futuristic  concepts, 
which  are  intended  to  keep  them  competitive  in  the  long-term,  instead 
of  just  near-term  products.  Disruptive  R&D  includes  significant 
technology  development  efforts  addressing  the  anticipated  customer 
needs  of  the  future,  potentially  leading  to  products  that  render 
competitive  products  irrelevant.  By  separately  organizing  disruptive 
R&D  from  incremental,  companies  are  able  to  protect  funds  from  the 
near-term  focused  business  unit  managers.  In  the  DOD  environment, 
budget  pressures  and  urgent  requirements  often  drive  military 
departments  to  focus  on  near-term  needs  over  long-term  innovation. 
For  instance,  a  2016  Air  Force  Studies  Board  report  found  that  in 
much  of  the  Air  Force,  little  or  no  space  for  innovation  exists.17 
Because  innovation  is  focused  on  future  needs,  the  report  found  that 
Air  Force  organizations  decide  they  can  wait  on  addressing  the  needs 
of  tomorrow.  The  report  found  that  across  the  Air  Force  as  a  whole, 
insufficient  processes  existed  to  support  “rapid-cycle”  innovation  with 
the  same  intensity  and  pace  Air  Force  personnel  regularly  bring  to 
bear  to  fulfill  other  missions. 


16GAO-06-883. 

1 7Air  Force  Studies  Board,  The  Role  of  Experimentation  Campaigns  in  the  Air  Force 
Innovation  Life  Cycle  (Washington,  D.C.:  2016). 
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•  Companies  fiercely  compete  with  one  another  for  customers  and  in 
ever-changing  market  conditions.  This  environment  requires  agility  in 
how  they  direct  their  technology  and  product  investments,  which 
includes  rapidly  initiating  new  projects  and  truncating  underperforming 
ones.  DOD,  on  the  other  hand,  operates  under  different  conditions.  Its 
budget  environment  may  incentivize  starting  and  sustaining  programs 
rather  than  discontinuing  underperforming  ones.  In  April  2014,  we 
found  that  budgets  to  support  major  acquisition  program  commitments 
must  be  approved  well  ahead  of  when  the  information  needs  to 
support  the  decision  is  available.18  DOD’s  S&T  community  operates 
under  similar  pressures  and  incentives  as  its  acquisition  community. 

In  this  environment,  we  found  that  it  is  easier  to  sustain  a  program 
until  its  funding  expires,  even  if  technical  performance  is  lacking. 
According  to  DOD  S&T  officials,  current  budgeting  and  funding 
processes  restrict,  rather  than  encourage,  innovation.  DOD  S&T 
executives  told  us  that  they  want  more  flexibility  outside  of  the 
cumbersome  2-year  PPBE  process  to  initiate  and  discontinue 
projects. 

•  Companies  set  their  own  budgets  internally  for  various  activities, 
including  R&D.  Conversely,  as  a  government  agency,  DOD  can 
influence,  but  not  set,  its  annual  budget.  Ultimately,  Congress 
determines  what  level  of  funding  to  appropriate  DOD,  including  for 
S&T-related  activities. 


New  Office  to  Be 
Responsible  for  Promoting 
Innovation  and 
Establishing  S&T  Policies 


Overcoming  many  of  these  challenges  may  ultimately  be  the 
responsibility  of  the  yet-to-be-created  Office  of  the  Under  Secretary  of 
Defense  for  Research  and  Engineering  (USD(R&E)).  The  National 
Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2017  calls  for  the 
establishment  of  the  position  of  USD(R&E)  to  serve  as  the  CTO  and 
elevate  and  enhance  the  mission  of  defense  technological  innovation.19 
This  office  will  have  greater  responsibilities  than  the  current  ASD(R&E) 
and  will  focus  on  innovation,  oversight,  and  policy  for  defense  research 
and  engineering,  technology  development  and  transition,  prototyping  and 
experimentation,  and  testing  activities.  Specifically,  where  ASD(R&E)  has 
taken  a  more  hands-off  approach  to  developing  S&T  policy,  Congress 
legislated  that  the  new  office  take  a  larger  role  in  establishing  policies  to 


18GAO,  Defense  Acquisitions:  Addressing  Incentives  is  Key  to  Further  Reform  Efforts , 
GAO-1 4-563T  (Washington,  D.C.:  Apr.  30,  2014). 

19Pub.  L.  No.  114-328  §  901(a),  which  amends  10  U.S.C.  by  striking  §  133  and  adding 
new  section  133a.  See  also,  H.  R.  Conf.  No.  114-840  atl  129-1 131  (2016). 
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overcome  the  challenges  DOD  currently  faces  with  promoting  innovation. 
The  USD(R&E)  will  also  be  responsible  for  the  allocation  of  resources  for 
defense  research  and  engineering,  and  unifying  these  efforts  across 
DOD.  The  fiscal  year  2017  NDAA  requires  this  position  to  be  created  by 
February  201 8.  In  March  2017,  DOD  reported  that  it  would  submit  final 
plans  for  creating  this  position  to  Congress  no  later  than  August  1, 2017, 
as  required  by  law. 


Conclusions 


DOD’s  S&T  investments  are  key  to  maintaining  our  nation’s  technological 
superiority  over  our  adversaries.  Congress  has  raised  questions  about 
DOD  being  innovative  enough  to  maintain  future  technology  superiority. 
The  business  imperatives  that  world-class  technology  companies  must 
operate  under  force  them  to  manage  their  S&T  portfolios  and  projects  to 
produce  better  outcomes  for  evolving  current  products,  as  well  as  well  as 
develop  disruptive  technologies  for  the  future.  Leading  companies  have 
shown  they  do  this  by  organizing  and  funding  R&D  to  avoid  the  pressures 
to  focus  on  incremental  innovation  at  the  expense  of  maintaining  their 
technological  edge  in  the  future.  With  its  focus  on  meeting  warfighter 
needs,  DOD  does  not  operate  under  similar  business  imperatives;  it  has 
not,  at  a  department-wide  level,  emphasized  the  need  to  invest  in 
disruptive  technologies.  Instead,  each  military  department’s  S&T 
organizational  construct  and  funding  processes  increase  emphasis  on 
investing  in  technologies  that  will  support  the  near-term  requirements  of  a 
major  weapon  system  acquisition  program  at  the  expense  of  investing  in 
innovative  technologies  that  are  not  linked  to  a  requirement. 

As  DOD  determines  the  roles  and  responsibilities  for  its  new  Under 
Secretary  of  Defense  for  Research  and  Engineering,  it  is  uniquely 
positioned  to  rethink  its  policies  that  govern  technology  development. 
While  it  may  not  be  practical  for  each  military  department  to  organize  its 
technology  development  as  leading  companies  do,  there  are  pockets 
within  each  department  that  are  implementing  some  aspects  of  leading 
company  practices.  However,  more  needs  to  be  done  to  facilitate  more 
systematic  adoption  of  these  practices  across  DOD.  Doing  so  can 
position  DOD  to  develop  more  innovative,  disruptive  technologies.  By  not 
taking  steps  to  ensure  the  right  balance  of  incremental  and  disruptive 
technology  investments,  DOD  lacks  visibility  into  whether  the 
technologies  it  is  developing  will  provide  superior  capabilities  to  counter 
future  and  emerging  adversarial  threats.  Additionally,  the  limited 
collaboration  with  product  developers,  limited  use  of  existing  flexible 
approaches  to  fund  S&T  projects  outside  of  the  2-year  planning  process, 
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and  limited  advanced  prototyping  of  new  technologies  by  the  labs  creates 
added  barriers  to  innovation. 


RprnmmpnrlatinnQ  fnr  To  ensure  that  DOD  is  positioned  to  counter  both  near  and  far  term 
UUU  U  Ucll  U  b  U  threats,  consistent  with  its  S&T  framework,  we  recommend  that  the 

Executive  Action  Secretary  of  Defense  direct  the  new  Under  Secretary  of  Defense  for 

Research  and  Engineering  annually  take  the  following  two  actions: 

•  define  the  mix  of  incremental  and  disruptive  innovation  investments 
for  each  military  department,  and 

•  assess  whether  that  mix  is  achieved. 

To  ensure  that  DOD  is  positioned  to  more  comprehensively  implement 
leading  practices  for  managing  science  and  technology  programs,  we 
recommend  that  the  Secretary  of  Defense  direct  the  new  Under  Secretary 
of  Defense  for  Research  and  Engineering  to  define,  in  policy  or  guidance, 
an  S&T  management  framework  that  includes  the  three  following  actions: 

•  emphasizes  greater  use  of  existing  flexibilities  to  more  quickly  initiate 
and  discontinue  projects  to  respond  to  the  rapid  pace  of  innovation; 

•  incorporates  acquisition  stakeholders  into  technology  development 
programs  to  ensure  they  are  relevant  to  customers;  and 

•  promotes  advanced  prototyping  of  disruptive  technologies  within  the 
labs  so  the  S&T  community  can  prove  these  technologies  work  to 
generate  demand  from  future  acquisition  programs. 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  draft  of  this  report  to  the  DOD  for  review  and  comment. 
DOD’s  written  comments  are  reprinted  in  appendix  III  of  this  report  and 
summarized  below. 

In  its  comments,  DOD  did  not  concur  with  each  of  our  recommendations, 
citing  that  it  is  premature  to  get  ahead  of  the  Secretary  of  Defense’s  final 
decisions  on  the  role  of  the  new  Under  Secretary  of  Defense  for 
Research  and  Engineering  (USD(R&E))  until  that  position  is  established, 
which  is  required  by  no  later  than  February  1, 2018.  We  believe,  however, 
as  the  roles  and  responsibilities  of  the  USD(R&E)  are  in  the  process  of 
being  deliberated,  that  it  is  appropriate  and  timely  for  the  Secretary  of 
Defense  to  ensure  that  the  USD(R&E)  be  responsible  for  implementing 
our  recommendations.  Although  it  did  not  concur,  DOD  identified  actions 
that  it  could  take  that  are  generally  responsive  to  our  recommendations. 
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Specifically,  in  response  to  our  recommendations  that  the  Secretary  of 
Defense  direct  the  USD(R&E)  to  define  and  assess  the  mix  of 
incremental  and  disruptive  innovation  investments  for  each  military 
department,  DOD  stated  that  it  would  need  to  coordinate  with  each 
military  department  to  establish  appropriate  goals  for  those  investments. 
DOD  further  noted  that  it  could  assess  whether  that  mix  is  achieved 
during  its  annual  S&T  Strategic  Overview  meeting.  We  continue  to 
believe  that  such  actions  are  necessary  to  ensure  that  DOD  is  positioned 
to  counter  both  near  and  far  term  threats. 

In  response  to  our  recommendation  that  the  USD(R&E)  define  an  S&T 
framework  that  emphasizes  greater  use  of  flexibilities  to  more  quickly 
initiate  and  discontinue  projects  to  respond  to  the  rapid  pace  of 
innovation,  DOD  identified  the  Laboratory  Quality  Enhancement  Program 
as  an  activity  to  leverage  existing  flexibilities.  This  program — which  DOD 
implemented  in  response  to  the  National  Defense  Authorization  Act  for 
Fiscal  Year  2017 — requires  DOD  to  create  panels  of  experts  to  make 
recommendations  to  the  Secretary  of  Defense  on  matters  related  to  S&T 
policy  and  practices.  DOD,  however,  did  not  explain  how  this  program 
would  help  the  labs  make  greater  use  of  existing  flexibilities  to  initiate 
projects,  such  as  those  granted  under  Section  219  of  the  Duncan  Hunter 
National  Defense  Authorization  Act  for  Fiscal  Year  2009.  We  continue  to 
believe  that  greater  use  of  existing  authorities,  such  as  those  provided 
under  Section  219,  could  help  labs  to  more  quickly  initiate  projects 
outside  of  the  normal  planning  cycle,  which  can  take  nearly  two  years  for 
a  project  to  be  funded. 

In  response  to  our  recommendation  that  the  USD(R&E)  define  an  S&T 
framework  that  incorporates  acquisition  stakeholders  into  technology 
development  programs,  DOD  identified  that  it  expects  the  USD(R&E)  to 
provide  policy  and  guidance  that  will  include  increased  engagement  with 
acquisition  stakeholders.  We  continue  to  believe  that  enhancing 
collaboration  between  the  S&T  and  acquisition  communities  is  critical  to 
ensuring  that  technologies  in  development  will  be  relevant  to  potential 
customers. 

In  response  to  our  recommendation  that  the  USD(R&E)  define  an  S&T 
framework  that  promotes  advanced  prototyping  of  disruptive  technologies 
within  the  labs,  DOD  noted  the  benefits  of  prototyping  and  that  it  is  a 
critical  piece  of  the  larger  research  and  engineering  strategy.  It  did  not, 
however,  identify  if  any  such  strategy  would  be  revised  to  promote  earlier 
prototyping  so  the  S&T  community  can  prove  technologies  work  and 
generate  demand  from  future  acquisition  programs.  We  continue  to 


Page  44 


GAO-17-499  Innovation  Best  Practices 


believe  that  establishing  an  S&T  framework  that  emphasizes  prototyping 
outside  of  acquisition  programs  is  needed. 

Additionally,  in  response  to  our  recommendations  that  the  Secretary  of 
Defense  direct  the  USD(R&E)  to  define  the  three  elements  above  in  an 
S&T  management  framework,  DOD  also  noted  that  Reliance  21  is 
expected  to  continue  serving  as  the  overarching  framework  for  the  S&T 
joint  planning  and  coordination  process.  We  continue  to  believe,  however, 
that  this  framework  does  not  fully  address  our  recommendations  and  that 
further  actions,  such  as  those  they  outlined  above,  are  necessary  for 
DOD  to  ensure  it  is  positioned  to  more  comprehensively  implement 
leading  practices  for  managing  S&T. 


We  are  sending  copies  of  the  report  to  the  appropriate  congressional 
committees;  the  Secretary  of  Defense;  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics;  and  the  Secretaries  of  the 
Army,  Navy,  and  Air  Force;  and  to  the  eight  leading  companies  we 
interviewed  about  their  practices  for  this  report.  In  addition,  the  report  will 
be  available  at  no  charge  on  GAO’s  website  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact 
me  at  202-512-4841  or  sullivanm@gao.gov.  Contact  points  for  our  Offices 
of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last 
page  of  this  report.  GAO  staff  who  made  major  contributions  to  this  report 
are  listed  in  appendix  IV. 


Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
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List  of  Committees 


The  Honorable  John  McCain 
Chairman 

The  Honorable  Jack  Reed 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Thad  Cochran 
Chairman 

The  Honorable  Richard  J.  Durbin 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Kay  Granger 
Chairwoman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


We  used  a  case  study  approach  to  identify  leading  commercial 
companies’  research  and  development  (R&D)  practices.  We  selected  and 
visited  eight  large  (Fortune  500-listed)  companies  that  were  U.S. 
companies  or  equivalent  foreign  companies,  or  were  owned  by  those 
companies.  Our  primary  goal  was  to  select  large  companies — those  more 
comparable  with  the  size  of  the  Department  of  Defense  (DOD)  than 
smaller  ones — from  a  range  of  different  industries,  that  were  profitable 
and  that  received  two  or  more  industry  awards  or  other  recognition  for 
technology  innovation  since  2014. 

We  used  the  following  sources  to  identify  companies  that  have  received 
awards  or  other  recognition:  Boston  Consulting  Group,  PwC,  MIT 
Technology  Review,  American  Business  Awards  Gold  Stevie  awards,  and 
Thomson-Reuters.  These  organizations  either  provide  annual  lists  of 
leading  innovator  companies  or  select  top  innovator  companies  for 
awards  on  an  annual  basis.  These  groups  are  positioned  to  be 
knowledgeable  regarding  who  the  “leading  companies”  are  for  the 
purposes  of  the  case  study  selection  for  our  review.  We  used  available 
corporate  stock  information  at  Morningstar.com  to  determine  whether  a 
company  has  been  profitable,  which  is  an  indicator  of  their  degree  of 
success  in  their  science  and  technology  development  efforts.  Below  are 
descriptions  of  the  eight  companies  featured  in  this  report. 

•  Amazon.com  sells  consumer  electronics,  operates  retail  websites 
serving  over  100  countries,  and  provides  cloud-computing  services  to 
hundreds  of  thousands  of  organizations  in  1 90  countries  around  the 
world.  Labi  26  is  Amazon’s  inventive  research  and  development 
company  that  designs  and  engineers  high-profile  consumer 
electronics,  including  Kindle  Fire  tablets,  Fire  TV  and  Amazon  Echo. 
Amazon’s  recent  recognitions  include  being  included  among  the 
Boston  Consulting  Group’s  (BCG)  “Most  Innovative  Companies,” 
PwC’s  “10  Most  Innovative  Companies,”  Thomson-Reuters’s  “Top  100 
Global  Innovators”  and  MIT  Technology  Review’s  “50  Smartest 
Companies.” 

•  Ashland  Global  Holdings,  Inc.  is  a  global  chemicals  company  serving 
customers  in  a  wide  range  of  consumer  and  industrial  markets, 
including  adhesives,  architectural  coatings,  automotive,  construction, 
energy,  food  and  beverage,  personal  care,  and  pharmaceutical. 
Valvoline  Inc.,  a  leading  producer  and  retailer  of  automotive  lubricants 
that  made  the  first  trademarked  American  motor  oil,  was  an  Ashland 
subsidiary  at  the  time  we  met  with  Valvoline  company  representatives. 
Ashland  currently  owns  a  controlling  interest  in  Valvoline  after  it 
became  a  separate  public  company  in  2016.  Ashland’s  recent 
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recognitions  include  a  Bronze  Innovation  Zone  Functional  Ingredient 
Award  and  Ringier  Coatings  Technology  Innovation  award  and  a 
Composites  and  Advanced  Materials  Exposition  award  for 
“Unsurpassed  Innovation.” 

•  The  Dow  Chemical  Company  delivers  a  broad  range  of  technology- 
based  products  and  solutions  to  customers  in  175  countries.  Dow 
drives  innovations  that  extract  value  from  material,  polymer,  chemical 
and  biological  science  to  help  address  many  of  the  world’s  most 
challenging  problems,  such  as  the  need  for  fresh  food,  safer  and  more 
sustainable  transportation,  clean  water,  energy  efficiency,  more 
durable  infrastructure,  and  increasing  agricultural  productivity.  Dow’s 
recent  recognitions  include  receiving  multiple  R&D  100  awards,  and 
being  included  among  BCG’s  “Most  Innovative  Companies”  and 
Clarivate  Analytics  (formerly  Thomson-Reuters’s)  “Top  100  Global 
Innovators.” 

•  Honeywell  International,  Inc.  invents  and  commercializes  technologies 
that  address  some  of  the  world’s  most  critical  challenges  around 
energy,  safety,  security,  productivity  and  global  urbanization. 
Honeywell’s  Aerospace  division,  which  is  discussed  in  this  report,  is  a 
leading  provider  of  aircraft  engines,  integrated  avionics,  systems  and 
service  solutions,  and  related  products  and  services  for  aircraft 
manufacturers,  and  turbochargers  to  improve  the  performance  and 
efficiency  of  passenger  cars  and  commercial  vehicles.  Honeywell’s 
recent  recognitions  include  receiving  American  Business  Awards  for 
“Most  Innovative  Company”  and  “Most  Innovative  Technology 
Company,”  and  being  included  among  Thomson-Reuters’s  “Top  100 
Global  Innovators.” 

•  General  Motors  Co.  and  its  partners  produce  vehicles  in  30  countries, 
including  the  Chevrolet,  Cadillac,  Baojun,  Buick,  GMC,  Holden, 
Jiefang,  Opel,  Vauxhall  and  Wuling  brands.  GM  develops  innovative 
new  technologies  offering  vehicle  electrification,  autonomous  driving, 
vehicle  health  management,  and  alternative  fuel  usage.  GM’s  recent 
recognitions  include  an  Edison  Award  for  Automotive  Computing,  an 
Automotive  News  “Pace  Award,”  and  being  included  among  Fast 
Company’s  “World’s  10  Most  Innovative  Companies  of  Automotive.” 

•  International  Business  Machines  Corporation  (IBM)  develops  and 
markets  cognitive  systems,  or  computers  that  learn  through 
interactions  with  people  and  data,  as  well  as  enterprise  systems  and 
software.  IBM  also  provides  cloud  computing,  consulting  and 
information  technology  implementation  services.  IBM’s  recent 
recognitions  include  receiving  an  R&D  100  award,  and  being  included 
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among  BCG’s  “Most  Innovative  Companies”  and  MIT  Technology 
Review’s  “50  Smartest  Companies.” 

•  Qualcomm  is  a  leader  in  the  commercialization  of  digital 
communication  technologies,  including  Code  Division  Multiple  Access 
(CDMA),  and  Long  Term  Evolution  (LTE),  for  cellular  wireless 
communication  applications.  They  also  develop  and  commercialize 
numerous  technologies  used  in  handsets  and  tablets.  They  also  own 
intellectual  property  contributing  to  other  commercial  technologies  like 
wireless  local  area  network,  global  positioning  system,  near  field 
communication,  and  Bluetooth.  Qualcomm’s  recent  recognitions 
include  receiving  an  R&D  100  award,  and  being  included  among  MIT 
Technology  Review’s  “50  Smartest  Companies”  and  Thomson- 
Reuters’s  “Top  100  Global  Innovators.” 

•  Siemens  AG  is  one  of  the  world’s  largest  producers  of  electrification, 
automation,  and  digitalization  technologies.  Siemens’s  products 
include  gas,  steam,  and  wind  turbines,  integrated  power  plant 
solutions,  power  grid  systems,  building  technologies,  rail  technologies, 
medical  imaging  and  diagnostics,  and  other  systems  for  industrial  use. 
The  company’s  recent  recognitions  include  receiving  two  R&D  100 
awards  and  being  included  among  BCG’s  “Most  Innovative 
Companies.”  Siemens  is  also  category  leader  in  the  Dow  Jones 
Sustainability  Index  ranking,  with  100  out  of  100  points  for  innovation 
management. 

For  each  of  the  companies,  we  conducted  semi-structured  interviews  with 
senior  management  officials  and  other  company  representatives 
knowledgeable  about  research  and  development  activities  to  gather 
consistent  information  about  processes  and  practices  companies  use  to 
manage  technology  development.  In  particular,  we  discussed  their  (1) 
organizational  structure  and  management  culture,  (2)  R&D  portfolio 
management  and  investment  strategy,  (3)  R&D  project  management 
practices,  and  (4)  their  technology  transition  process,  including  when 
transition  occurs,  the  organizations  involved,  and  how  technology  is 
funded  throughout  the  transition  phase.  We  synthesized  each  company’s 
processes  and  created  summary  documents,  which  the  companies  then 
reviewed  for  accuracy  and  completeness  to  validate  our  assessment  of 
their  specific  practices.  Using  this  validated  information,  we  identified  the 
practices  that  were  consistent  among  the  selected  companies  and  which 
company  representatives  considered  key  to  promoting  innovation.  We 
also  presented  our  analysis  of  these  leading  practices  to  senior  DOD 
Science  and  Technology  (S&T)  executives  within  the  Office  of  the 
Secretary  of  Defense,  the  military  services,  and  other  defense  research 
organizations  to  obtain  their  views  on  the  practices. 
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To  identify  the  extent  to  which  DOD  can  employ  these  leading 
commercial  practices,  we  interviewed  officials  responsible  for  the 
management,  execution,  and  oversight  of  DOD’s  S&T  enterprise.  At  the 
Office  of  the  Secretary  of  Defense  and  military  department  headquarters 
level,  those  responsible  for  the  management  and  oversight  of  S&T 
activities,  we  met  with  officials  from  the 

•  Office  of  the  Assistant  Secretary  of  Defense  for  Research  and 
Engineering; 

•  Office  of  the  Deputy  Assistant  Secretary  of  the  Army  for  Research  and 
Technology; 

•  Office  of  the  Deputy  Assistant  Secretary  of  the  Air  Force  for  Science, 
Technology,  and  Engineering; 

•  Office  of  the  Deputy  Assistant  Secretary  of  the  Navy  for  Research, 
Development,  Test,  and  Evaluation;  and 

•  Office  of  Naval  Research. 

We  also  met  with  military  department  laboratory  officials  responsible  for 
the  management  and  execution  of  S&T  activities  from  the 

•  Army  Research  Laboratory; 

•  Army  Armament  Research,  Development,  and  Engineering  Center; 

•  Army  Engineer  Research  and  Development  Center; 

•  Air  Force  Research  Laboratory; 

•  Naval  Research  Laboratory;  and 

•  Naval  Undersea  Warfare  Center — Division  Newport. 

Finally,  we  met  with  officials  from  the  Defense  Advanced  Research 
Projects  Agency  (DARPA)  responsible  for  the  planning  and  oversight  of 
their  S&T  activities. 

We  conducted  semi-structured  interviews  at  each  laboratory  and  DARPA 
to  gather  consistent  information  about  processes  and  practices  these 
organization  used  to  manage  S&T  activities.  In  particular,  we  discussed 
their  (1)  organizational  structure  and  management  culture,  (2)  S&T 
portfolio  management  and  investment  strategy,  (3)  S&T  project 
management  practices,  and  (4)  their  technology  transition  process.  We 
compared  and  contrasted  those  practices  with  the  practices  identified 
through  our  meetings  with  leading  commercial  companies  to  determine 
the  extent  to  which  DOD  is  employing  these  practices.  Where 
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appropriate,  we  reviewed  relevant  regulations,  policies,  and  guidance  that 
establish  the  framework  for  how  DOD  S&T  organizations  plan,  budget 
and  execute  S&T  activities,  including  the  Assistant  Secretary  of  Defense 
for  Research  and  Engineering’s  Reliance  21  Operating  Principles  and 
DOD’s  Financial  Management  Regulation.  To  further  our  understanding 
of  the  S&T  management  practices  being  used  at  the  military  department 
labs,  we  reviewed  at  least  two  recent  S&T  projects  by  each  of  these  labs. 
These  projects  were  identified  by  lab  officials  and  included  projects 
deemed  successful  as  well  as  ones  identified  as  unsuccessful. 

Finally,  we  hosted  a  forum  of  DOD  S&T  executives  in  December  2016  to 
identify  potential  opportunities  for  DOD  to  adopt  leading  commercial 
practices  in  S&T  management,  as  well  as  any  barriers  to  adopting  these 
practices.  Forum  participants  included  the  following: 

•  Ms.  Mary  Miller,  Principal  Deputy  to  the  Assistant  Secretary  of 
Defense  for  Research  and  Engineering; 

•  Dr.  Melissa  Flagg,  Deputy  Assistant  Secretary  of  Defense  for 
Research 

•  Mr.  Michael  Holthe,  Acting  Director  of  Technology,  Office  of  the 
Deputy  Assistant  Secretary  of  the  Army  for  Research  and 
Technology; 

•  Dr.  David  Walker,  Deputy  Assistant  Secretary  of  the  Air  Force  for 
Science,  Technology,  and  Engineering; 

•  Dr.  Phil  Perconti,  Acting  Director,  Army  Research  Laboratory; 

•  Dr.  Jeff  Holland,  Director,  Army  Engineer  Research  and  Development 
Center; 

•  Mr.  Jyuji  Hewitt,  Executive  Deputy,  Army  Research,  Development, 
and  Engineering  Command; 

•  Mr.  John  Uscilowicz,  Director  Plans,  Programs,  Analysis,  and 
Evaluation,  Army  Medical  Research  and  Materiel  Command; 

•  Dr.  Morley  Stone,  Chief  Technology  Officer,  Air  Force  Research 
Laboratory; 

•  Dr.  Edward  Franchi,  Acting  Director,  Naval  Research  Laboratory; 

•  Dr.  Stephen  Russell,  Director  of  Science  and  Technology,  Space  and 
Naval  Warfare  Systems  Command;  and 

•  Mr.  Ellison  Urban,  Special  Assistant  to  the  Director,  Defense 
Advanced  Research  Projects  Agency. 
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Table  1:  Department  of  Defense  (DOD)  Research,  Development,  Test,  and  Evaluation  (RDT&E)  Budget  Activities 

DOD  RDT&E  Budget  Activity 

Description 

Science  and 
technology  funding 

Basic  research  (6.1) 

Scientific  study  and  experimentation  focusing  on  increasing  fundamental 
knowledge,  which  may  address  long-term  national  security  needs.  Includes  pre- 
Milestone  A  efforts. 

Applied  research  (6.2) 

Research  focuses  on  the  expansion  and  application  of  knowledge  and  is 
directed  toward  general  military  needs  to  determine  the  initial  feasibility  and 
practicality  of  proposed  solutions.  Includes  pre-Milestone  B  efforts. 

Advanced  technology 
development  (6.3) 

Concept  and  technology  demonstrations  that  assess  the  technological 
feasibility,  operability,  and  producibility  of  components,  subsystems,  or  system 
models.  Demonstrations  evaluate  general  military  utility  or  cost  reduction 
potential  of  the  technology.  Projects  in  this  category  should  have  the  goal  of 
moving  out  of  S&T  and  into  the  acquisition  process  within  5  years.  Includes  pre- 
Milestone  B  efforts  and  technologies  generally  have  a  Technology  Readiness 
Level  (TRL)  of  4,  5,  or  6. 

Acquisition-based 

funding 

Advanced  component 
development  &  prototypes 
(6.4) 

System  specific  evaluations  of  integrated  technologies,  representative  models, 
or  prototype  systems  in  a  realistic  operating  environment.  Focuses  on  proving 
component  and  subsystem  maturity  prior  to  integration  into  major  systems. 
Includes  pre-Milestone  B  efforts  and  TRL  6  or  7  should  be  achieved. 

System  development  & 
demonstration  (6.5) 

Engineering  and  manufacturing  development  tasks  aimed  at  meeting 
requirements  prior  to  full-rate  production.  Prototype  performance  is  near  or  at 
planned  operational  system  levels.  Conduct  live  fire  and  initial  operational  test 
and  evaluation.  Includes  post-Milestone  B  efforts  to  support  Milestone  C 
decisions. 

RDT&E  management  support 
(6.6) 

Efforts  to  sustain  and/or  modernize  installations  or  operations  required  for 

RDT&E  such  as  test  ranges,  military  construction,  and  studies  and  analyses  in 
support  of  RDT&E. 

Operational  system 
development  (6.7) 

Efforts  to  upgrade  systems  that  have  been  fielded  or  will  soon  enter  full  rate 
production.  Includes  post-Milestone  C  efforts. 

Source:  GAO  summary  of  DOD  regulations.  |  GAO-17-499 


Note:  Technology  Readiness  Levels  (TRL)  are  a  tool  that  DOD,  among  others,  uses  to  assess 
technology  maturity.  TRLs  are  measured  on  a  scale  from  1  to  9,  beginning  with  paper  studies  of  a 
technology’s  feasibility  and  culminating  with  a  technology  fully  integrated  into  a  completed  product. 
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RESEARCH 
AND  ENGINEERING 


ASSISTANT  SECRETARY  OF  DEFENSE 
3030  DEFENSE  PENTAGON 
WASHINGTON,  DC  2030 1 -3030 


JUN  1  5  2017 


Mr.  Michael  J.  Sullivan 

Director.  Acquisition  and  Sourcing  Management 
U.S.  Go\emment  Accountability  Office 
441  C.  Street.  NW 
Washington  DC  20548 

Dear  Mr.  Sullivan: 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report 
GAO- 17-499.  "DEFENSE  SCIENCE  AND  TECHNOLOGY:  Adopting  Best  I’raetices 
Can  Improve  Innovation  Investments  and  Management,"  dated  May  12.  2017.  (GAO 
Code  100656). 

The  Department  intends  to  take  full  adv  antage  of  the  flexibility  inherent  in  section 
901  of  the  National  Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2017  on  the 
review  and  recommended  organizational  and  management  structure  for  the  Office  of  the 
Secretary  of  Defense  (  OSD  ).  DoD  nonconcurs  with  all  recommendations  on  die  basis  of 
the  on-going  reorganization  effort  as  the  position  of  the  Under  Secretary  of  Defense  for 
Research  and  Engineering  (USD  (R&E))  will  become  effective  not  later  than  1  February 
2018.  While  it  would  he  premature  to  get  ahead  of  the  Secretary's  final  decisions  until 
the  reorganization  of  the  Office  of  the  Secretary  of  Defense  is  complete  and  positions  are 
established,  our  response  describes  the  general  direction  of  the  DoD's  efforts  addressing 
the  recommendations  of  the  report. 


My  point  of  contact  is  COL  Angel  Nievesortiz  who  can  be  reached  at 
angel.Enievesortiz.mil  a 'mail.mil  and  phone  703-695-0598. 


Enclosure 
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GAO  DRAFT  REPORT  DATED  MAY  1 1, 2017 
GAO-17-499  (GAO  CODE  100656) 

“DEFENSE  SCIENCE  AND  TECHNOLOGY:  ADOPTING  BEST  PRACTICES  CAN 
IMPROVE  INNOV  ATION  INVESTMENTS  AND  MANAGEMENT" 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 

RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense.  ..direct  the 
new  Under  Secretary  of  Defense  for  Research  and  Engineering  to  annually:  define  the  mix  of 
incremental  and  disruptive  innovation  investments  for  each  military  department,  and 

Doll  RESPONSE:  Nonconcur.  I  he  position  of  the  I  Inder  Secretary  of  Defense  for  Research 
and  Engineering  (USD  (R&E))  will  become  effective  not  later  than  1  February  2018.  Any 
recommendations  or  anticipated  actions  by  the  l  hidcr  Secretary  are  premature  until  the 
reorganization  of  the  Office  of  the  Secretary  of  Defense  is  complete  and  positions  are 
established.  I  he  report  highlights  the  challenges  of  the  Planning.  Programming,  Budgeting  and 
Execution  (PPBE)  process  and  the  Department’s  culture  in  the  management  of  the  science  and 
technology  portfolio.  As  the  Secretary  operationalizes  the  roles,  responsibilities  and  authorities 
of  the  USD  (R&E),  the  DoD  components  are  expected  to  retain  significant  authority  in  the 
management  of  their  science  and  technology  portfolios.  Coordination  w  ith  the  DoD  components 
is  necessary  to  establish  appropriate  goals  in  the  mix  of  incremental  and  disruptive  innovation 
investments  in  their  portfolios  based  on  the  requirements  of  their  military  departments. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense...  direct  the 
new  Under  Secretary  of  Defense  for  Research  and  Engineering  to  annually:  assess  whether  that 
mix  is  achieved. 

DoD  RESPONSE:  Nonconcur.  The  position  of  the  Linder  Secretary  of  Defense  for  Research 
and  Engineering  (USD  ( R&E))  will  become  effective  not  later  than  I  February  201 8.  Any 
recommendations  or  anticipated  actions  by  the  Under  Secretary  are  premature  until  the 
reorganization  of  the  Office  of  the  Secretary  of  Defense  is  complete  and  positions  are 
established.  This  assessment  on  the  mix  of  incremental  and  disruptive  innovation  investments 
for  each  military  department  can  be  incorporated  into  the  annual  Science  and  Technology  (S&T| 
Strategic  Overview  meeting,  rhis  meeting  is  an  annual  event  where  the  DoD  S&T  leadership 
discuss  their  organizations'  priorities,  portfolio  content,  and  strategic  direction.  It  is  a  central 
element  of  the  Department's  process  for  ensuring  S&T  alignment  with  corporate  directions  and 
priorities. 

RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense...  direct  the 
new  Under  Secretary  of  Defense  for  Research  and  Engineering  to  define,  in  policy  or  guidance, 
an  S&T  management  framework  that  does  the  following:  emphasizes  greater  use  of  existing 
flexibilities  to  more  quickly  initiate  and  discontinue  projects  to  respond  to  the  rapid  pace  of 
innovation: 
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DoD  RESPONSE:  Nonconcur.  The  position  of  the  Under  Secretary  of  Defense  for  Research 
and  Engineering  (USD  <  R&E ))  will  become  effective  not  later  than  1  February  2018.  Any 
recommendations  or  anticipated  actions  by  the  Under  Secretary  are  premature  until  the 
reorganization  of  the  Office  of  the  Secretary  of  Defense  is  complete  and  positions  are 
established.  As  the  Secretary'  operationalizes  the  roles,  responsibilities  and  authorities  of  the 
USD  (R&U).  it  is  expected  that  Reliance  21  will  continue  as  the  overarching  framework  for  the 
science  and  technology  (S&T)  joint  planning  and  coordination  process.  This  framework  ensures 
a  collective  understanding  of  the  priorities,  requirements,  and  opportunities  of  the  DoD 
organizations  that  manage  critical  S&T  resources.  It  is  effective  in  continuously  improving 
support  to  the  warfighter  by  propagating  and  coordinating  an  understanding  of  military  capability 
gaps  and  defense  needs  throughout  the  S&T  community.  The  l  aboratory  Quality  Enhancement 
Program  (I..QEP)  is  another  existing  activity  to  leverage  existing  flexibilities  granted  by 
legislation  and  enhance  coordination  across  the  laboratory  enterprise  including  sharing  of  best 
practices  to  improve  the  ability  of  the  Department  to  respond  to  the  rapid  pace  of  innovation. 

RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense...  direct  the 
new  Under  Secretary  of  Defense  for  Research  and  Engineering  to  define,  in  policy  or  guidance, 
an  S&T  management  framework  that  does  the  following:  incorporate  acquisition  stakeholders 
into  technology  development  programs  to  ensure  they  are  relevant  to  customers:  and 

DoD  RESPONSE:  Nonconcur.  The  position  of  the  Under  Secretary'  of  Defense  for  Research 
and  Engineering  (USD  (R&E))  will  become  effective  not  later  titan  I  February  2018.  Any 
recommendations  or  anticipated  actions  by  the  I  nder  Secretary  are  premature  until  the 
reorganization  of  the  Office  of  the  Secretary  of  Defense  is  complete  and  positions  are 
established.  As  the  Secretary  operationalizes  the  roles,  responsibilities  and  authorities  of  the 
U  SD  (R&E),  it  is  expected  this  position  will  provide  policy  and  guidance  on  technology 
development,  prototyping,  experimentation,  and  technology  transition,  which  will  include 
increased  engagement  with  acquisition  stakeholders. 

RECOMMENDATION  S:  The  GAO  recommends  that  the  Secretary  of  Defense. . ,  direct  the 
new  Under  Secretary  of  Defense  for  Research  and  Engineering  to  define,  in  policy  or  guidance, 
an  S&T  management  framework  that  does  the  following:  promotes  advanced  prototyping  of 
disruptive  technologies  within  the  labs  so  that  S&  I  community  can  prove  these  technologies 
work  to  generate  demand  from  future  acquisition  programs. 

DoD  RESPONSE:  Nonconcur.  The  position  of  the  Under  Secretary  of  Defense  for  Research 
and  Engineering  (USD  (R&EV)  will  become  effective  not  later  than  I  February  2018.  Any 
recommendations  or  anticipated  actions  by  the  Under  Secretary  are  premature  until  the 
reorganization  of  the  Office  of  the  Secretary  of  Defense  is  complete  and  positions  are 
established.  The  ability  of  prototyping  to  evaluate  and  reduce  technical  risk  and  clarify  the 
resource  picture  that  drives  costs  makes  prototyping  a  critical  piece  of  the  larger  research  and 
engineering  strategy.  Prototyping  will  also  help  DoD  validate  and  finalize  requirements, 
improve  manufacturing  techniques,  evaluate  new  concepts,  and  field  initial  quantities  of  new 
systems  faster. 
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Appendix  IV:  GAO  Contact  and  Staff 
Acknowledgments 


GAO  Contact 

Michael  J.  Sullivan,  (202)  512-4841,  sullivanm@gao.gov 

In  addition  to  the  contact  named  above,  Christopher  R.  Durbin,  Assistant 
Director;  Marie  Ahearn;  Emily  Bond;  Jared  Dmello;  Lorraine  Ettaro;  Rich 
Hung;  Justin  Jaynes;  Ron  La  Due  Lake;  Sean  Seales;  Brian  Smith;  and 
Robin  Wilson  made  significant  contributions  to  this  report. 


Staff 

Acknowledgments 


(100656) 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and  investigative 
arm  of  Congress,  exists  to  support  Congress  in  meeting  its  constitutional 
responsibilities  and  to  help  improve  the  performance  and  accountability  of  the 
federal  government  for  the  American  people.  GAO  examines  the  use  of  public 
funds;  evaluates  federal  programs  and  policies;  and  provides  analyses, 
recommendations,  and  other  assistance  to  help  Congress  make  informed 
oversight,  policy,  and  funding  decisions.  GAO’s  commitment  to  good  government 
is  reflected  in  its  core  values  of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  GAO’s  website  (http://www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  website  newly  released  reports,  testimony,  and  correspondence.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products,  go  to  http://www.gao.gov 
and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of  production  and 
distribution  and  depends  on  the  number  of  pages  in  the  publication  and  whether 
the  publication  is  printed  in  color  or  black  and  white.  Pricing  and  ordering 
information  is  posted  on  GAO’s  website,  http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card,  MasterCard, 

Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Linkedln,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 

Visit  GAO  on  the  web  at  www.gao.gov  and  read  The  Watchblog. 

To  Report  Fraud, 

Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512-4400, 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125, 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 

Washington,  DC  20548 

Strategic  Planning  and 
External  Liaison 

James-Christian  Blockwood,  Managing  Director,  spel@gao.gov,  (202)  512-4707 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7814, 
Washington,  DC  20548 
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